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ABSTRACT 


T.iis  is  Volume  II  of  a  three-volume  final  report  that  covers 
Phase  II  of  a  three-phase  project  on  the  Use  of  Air  Force  ADP  Expe¬ 
rience  to  Assist  Air  Force  ADP  Management.  In  Phase  I,  a  feasible 
concept  and  preliminary  approach  to  using  experience  was  synthesized; 
in  Phase  II,  the  approach  was  refined,  the  concept  was  validated,  and 
the  potential  use  of  experience  was  broadened;  and  in  Phase  III,  the 
improved  and  expanded  approach  will  be  implemented  Air  Force-wide. 

Volume  I  of  the  final  report  covers  the  following:  the  history  of 
the  project;  conclusions  of  Phase  II  and  recommendations  for  Phase 
ILL;  and  summaries  of  Phase  II  activities,  the  Phase  III  concept  and 
plan,  and  the  pilot  version  of  the  ADP  Experience  Handbook  and  Primer. 
Volume  II  reviews  the  four  major  activities  of  Phase  II:  data  collection, 
data  analysis,  ADP  Experience  Handbook  development,  and  Phase  III 
planning.  Volume  HI  presents  the  detailed  Phase  HI  operational  con¬ 
cept  and  development  plan,  followed  by  a  summary  of  cost  and  benefits. 

This  is  Volume  II,  in  which  the  four  major  activities  of  Phase  II 
are  described.  The  design  of  the  data  collection  questionnaire  was  based 
on  the  ./UDPS  model  (a  concept  of  a  "total"  ADPS)  and  the  workload  model 
representing  attributes  of  an  ADPS.  Data  were  collected  on  a  stratified 
18-ADPS  sample,  and  the  statistical  analysis  of  these  data  produced  five 
cost  estimation  equations.  In  addition,  the  data  were  used  to  produce  a 
seven-page  system  description  of  each  ADPS,  which  became  the  core  of 
the  ADP  Experience  Handbook.  A  Phase  IH  operational  concept  and  de¬ 
velopment  plan  was  also  synthesized. 
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I.  INTRODUCTION 


This  is  Volume  II  of  a  three-volume  final  report  that  marks  the 
completion  by  Planning  Research  Corporation  of  a  research  study  on  the 
Use  of  Air  Force  ADP  Experience  to  Assist  Air  Force  ADP  Management. 
The  study  is  the  second  phase  of  a  three-phase  project;  Phase  II  is  to 
validate  and  refine  concepts  developed  in  Phase  I  and  to  develop  an  opera¬ 
tional  concept  and  plan  for  implementation  in  Phase  III. 

The  purpose  of  the  final  report  is  to  present  the  objectives,  activi¬ 
ties,  findings,  and  conclusions  of  Phase  II  and  to  submit  an  operational 
concept  and  development  plan  for  Phase  III.  These  are  reported  in  Vol¬ 
ume  II  and  Volume  III,  respectively.  In  addition,  the  pilot  version  of  the 
ADP  Experience  Handbook  and  a  Primer  that  serves  as  an  elementary 
text  for  training  potential  users  of  the  handbook  are  produced  as  two  sep¬ 
arate  volumes  distinct  from  this  final  report  (refer  to  PRC  documents 
R-930  and  R-931).  Volume  I  provides  a  concise  summary  of  Volumes  II 
and  III,  and  a  brief  description  of  the  ADP  Experience  Handbook  and 
Primer . 

The  purpose  of  Volume  II  is  to  present  the  objectives,  activities, 
findings,  and  conclusions  of  Phase  II  in  detail.  This  volume  is  directed 
to  those  audiences  that  desire  a  complete  description  of  all  or  any  part 
of  the  activities  in  Phase  II.  Volume  I  provides  a  concise  summary  of 
Volumes  II  and  III  and  a  brief  description  of  the  ADP  Experience  Hand¬ 
book  and  Primer. 

This  volume  is  organized  into  four  major  sections  covering  the 
major  activities  of  Phase  II:  data  collection,  data  analysis,  experience 
handbook  development,  and  Phase  III  planning.  The  section  on  data  col¬ 
lection  covers  model  development,  ADPS  sample,  data  collection,  and 
data  reduction.  The  section  on  data  analysis  deals  with  refinement  of 
the  workload  model,  testing  for  subpopulations,  and  derivation  of  the 
cost  estimation  equations.  The  section  on  experience  handbook  develop¬ 
ment  reviews  the  development  of  cost  estimation  graphs,  system  descrip¬ 
tions,  development  of  indexes,  and  construction  of  the  Primer.  The  sec¬ 
tion  on  Phase  III  planning  discusses  the  development  of  the  Phase  III 
operational  concept  and  plan.  Seven  appendixes  support  the  text  with 
data  and  procedures. 

To  aid  the  reader,  especially  for  the  data  collection  and  data  analy¬ 
sis  sections,  a  brief  classification  and  definition  of  terminology  associated 
with  models  and  variables  as  used  in  this  volume  will  be  given  here.  The 
relationships  among  the  various  terms  are  shown  in  Figure  1. 

The  dependent  variables  are  classified  into  planning  factors,  and  the 
independent  variables  are  classified  into  estimating  factors.  The  depend¬ 
ent  variables  are  to  be  estimated  by  the  independent  variables,  which  are 
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then  called  predictors.  The  independent  variables  also  are  referred  to 
as  workload  descriptors  and  explanatory  variables.  The  workload  model 
consists  of  the  entire  set  of  variables,  both  dependent  and  independent. 
The  cost  model  is  a  subset  of  the  workload  model,  consisting  of  all  the 
cost  variables  that  comprise  the  total  cost  of  an  ADPS  and  the  workload 
descriptors  that  are  causally  related  to  each  of  the  cost  variables.  The 
regression  model  is  a  subset  of  the  cost  model  that  is  used  in  regression 
analysis;  the  inter  cor  related  workload  descriptors  for  each  cost  variable 
in  the  cost  model  has  been  removed. 

The  ADPS  model  is  the  concept  of  the  total  ADPS  and  is  used  as  a 
basis  for  data  collection  and  system  description.  The  terms  "macro¬ 
description,  "  "total  description,  "  and  "system  description"  are  used 
synonymously  when  referring  to  an  ADPS. 
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FIGURE  1  -  WORKLOAD  MODEL 


II.  DATA  COLLECTION 


The  purpose  of  this  section  is  to  review  the  data  collection  activi¬ 
ties  of  Phase  II. 

A.  Objectives 


The  objectives  of  data  collection  for  Phase  II  were  twofold: 

o  To  collect  sufficient  descriptive  information  to  permit  mac- 
rodescriptions  of  selected  Air  Force  ADPS. 

o  To  collect  sufficient  numerical  cost  data  and  workload  de¬ 

scriptor  parameters  to  permit  the  development  of  cost  es¬ 
timating  relationships  for  ADP  systems. 

To  achieve  these  objectives,  the  following  tasks  were  performed: 

o  Development  of  an  ADPS  model 

o  Redefinition  of  the  workload  model 

o  Development  of  operational  definitions  and  measures  for 

variables 

o  Restructure  of  ADPS  sample 

o  Development  of  data  collection  procedures 

o  Reduction  of  collected  data 

Each  of  these  tasks  is  described  in  the  subsequent  sections. 
Finally,  a  section  covering  findings  completes  the  section  on  data 
collection. 

B.  ADPS  Model 


In  order  to  collect  sufficient  descriptive  information  to  produce  a 
macrodescription  of  an  ADPS,  a  model  representing  the  concept  of  a 
to:al  ADPS  was  developed.  This  model  served  as  a  basis  for  and  guided 
the  development  of  the  following  major  activities  in  data  collection: 

o  Design  of  the  questionnaire 

o  Collection  of  data  during  trips 

o  Compilation  and  reduction  of  first-level  data 

o  Preparation  of  system  summaries  for  the  midpoint  report 
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o  Preparation  of  system  descriptions  for  the  pilot  version  of 
the  ADP  Experience  Handbook 

The  principal  objective  in  the  development  of  the  ADPS  model  was 
to  create  a  concept  with  the  following  characteristics: 

o  Logical  breakdown  for  organization  of  the  interviewing 
activity 

o  Ease  of  explanation  to  and  understanding  by  the  interviewee 
of  the  concept 

o  Organization  along  lines  of  information  availability 

o  Compatibility  with  many  forms  of  Air  Force  ADP  systems 

The  concept  developed  for  describing  the  total  ADPS  was  based  on 
the  evolution  of  activities  of  the  ADPS  over  time.  The  time  axis  for  the 
ADPS  was  divided  into  four  major  periods.  These  were  called  Proposal 
Phase,  Development  Phase,  Operations  Phase,  and  Future  Plans.  These 
phases  were  not  always  clearcut,  but,  for  the  purposes  of  the  Phase  II 
study  only,  they  were  defined  as  follows: 

o  Proposal  Phase:  This  covers  the  period  from  the  conception 
of  the  system  to  the  time  the  proposal  for  the  system  was 
approved 

o  Development  Phase:  This  covers  the  period  from  the  ap- 
proval  of  the  proposal  or  the  beginning  of  system  design  to 
the  time  when  the  system  was  declared  operational 

o  Operations  Phase:  This  covers  the  period  from  the  time  the 
systems  was  declared  operational  to  the  present  time 

o  Future  Plans:  This  covers  the  period  beyond  the  present 
time 

See  Table  1  for  a  schematic  representation  of  the  total  ADPS  concept. 
Within  each  phase,  the  types  of  data  of  major  interest  are  itemized. 

C.  Workload  Model 


A  workload  model  was  defined  in  Phase  I.  It  was  hypothesized 
that  the  key  to  retrieving  experience  information  was  workload  - -quanti¬ 
tative  measures  of  the  information  processed.  The  reasons  for  using 
workload  rather  than  some  other  factors  were  as  follows: 

o  Workload  is  a  direct  causal  factor  for  cost  and  development 
time 
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o  Workload  is  amenable  to  quantitative  measurement 

o  Workload  should  be  available  in  a  proposal  for  an  ADPS 

Forty  numerical  workload  descriptors  were  advanced  in  Phase  I 
as  those  satisfying  the  three  criteria.  These  workload  descriptors 
were  to  be  analyzed  and  evaluated  during  Phase  II  by  statistical  tech¬ 
niques  on  sampled  ADP  systems  (1)  to  determine  relationships  between 
ADPS  workload  descriptors  and  ADPS  cost  and  development  time,  and 
(Z)  to  refine  those  relationships  to  a  well-defined,  sensitive,  and  small 
set  of  workload  descriptors. 

During  the  initial  stages  of  Phase  II,  the  workload  model  served 
two  major  functions.  Firstly,  the  design  of  the  data  collection  ques¬ 
tionnaire  was  based  on  the  ADPS  model  and  the  workload  model.  And 
secondly,  the  relevant  causal  factors  for  use  in  the  regression  analy¬ 
sis  to  derive  cost  estimating  relationships  were  obtained  from  the  work¬ 
load  model. 

During  the  de  sign  of  the  que  stionnair  e,  the  or  iginal  workload  model 
was  modified  and  expanded.  Sub  sequent  to  data  collection,  this  model  was 
further  refined,  some  variables  were  dropped,  and  others  were  com¬ 
bined.  (See  subsection  II. D.)  The  modifications  and  refinements  were 
neces  sitated  by  the  unavailability  of  data  for  some  variables.  The  re¬ 
sulting  workload  model  is  schematically  depicted  in  Figure  Z,  and  its 
function  is  described  below. 

The  workload  model  became  the  basis  for  development  of  the  cost 
model.  The  cost  model  is  comprised  of  a  set  of  dependent  variables 
called  cost  factors,  which  together  represented  the  total  cost  for  devel¬ 
opment  and  operations  of  an  ADPS.  (See  Table  4.  )  The  develop¬ 
ment  of  the  cost  model  is  described  in  subsection  III.B.6. 

The  usefulness  of  the  workload  model  for  deriving  cost  estimating 
relationships  depended  entirely  on  availability  of  historical  data  for  those 
sets  of  variables  that  represent  the  characteristics,  functions,  and  costs 
of  the  sampled  ADPS.  The  set  of  workload  descriptors  are  the  independ¬ 
ent,  variables  or  estimating  factors  in  the  cost  model,  and  will  later  be 
used  to  derive  the  cost  estimating  relationships  by  a  statistical  tech¬ 
nique  called  regression  analysis. 

The  workload  factor  is  that  set  of  independent  variables  that  re¬ 
late  to  the  inputs,  outputs,  and  data  base  functions  of  the  workload  model. 
The  complexity  factor  relates  to  the  processing  functions,  the  education 
and  experience  factor  relates  to  personnel,  and  the  machine  maturity 
factor  relates  to  equipment.  Each  of  the  independent  variables  is  also 
referred  to  in  this  report  as  a  workload  descriptor. 

D  Operational  Definitions  and  Measures  of  Workload  Descriptors 


The  value  of  data  analysis  is  heavily  dependent  on  the  accuracy 
and  uniformity  of  the  data  collected.  The  accuracy  and  uniformity  of 
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TABLE  1  -  ADPS  MODEL  FOR  PHASE  II  DATA  COLLECTION -- 
CONCEPT  OF  TOTAL  ADPS 
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FIGURE  2  -  WORKLOAD  MODEL  (INDEPENDENT  VARIABLES- -ESTIMATING  FACTORS) 


the  observations,  in  turn,  are  dependent  to  a  large  degree  on  the  pre¬ 
ciseness  of  definition  of  the  variables  in  the  statistical  analysis. 

During  Phase  I,  a  preliminary  workload  model  was  described  and 
workload  descriptors  defined.  At  the  beginning  of  Phase  II,  the  defini¬ 
tions  of  these  descriptors  were  reviewed  and  redefined  during  the  proc¬ 
ess  of  constructing  the  questionnaire.  Following  the  first  data  collec¬ 
tion  trip  (see  subsection  II. F. 4),  the  experience  gained  was  used  to 
modify  and  enhance  the  earlier  definitions.  In  addition,  several  vari¬ 
ables  were  dropped  and  a  number  of  variables  were  added.  These  were 
categorized  into  57  independent  variables  (estimating  factor s)  and  30  de¬ 
pendent  variables  (planning  factors).  See  Numerical  Data  Summary 
Sheet  in  Appendix  B. 

The  second  and  third  data  collection  trips  provided  additional  ex¬ 
perience  on  the  form  of  data  that  were  collectible.  The  wide  variety  of 
systems  encountered  helped  to  shake  down  and  test  the  adequacy  of  def¬ 
inition  of  variables.  From  this  experience,  the  definition  of  each  vari¬ 
able  was  scrutinized  and  redefined  when  necessary.  The  final  list  of 
variables  used  in  the  data  analysis  and  their  operational  definitions  can 
be  found  in  Appendix  C.  These  include  26  independent  variables,  which 
constitute  the  preliminary  set  of  estimating  factors,  and  7  dependent 
variables,  which  constitute  the  preliminary  set  of  planning  factors.  The 
discussion  of  data  reduction  (subsection  II. G)  will  describe  how  this  set 
was  obtained. 

E.  ADPS  Sample 


On  24  February  1966,  project  personnel  discussed  with  AFADA 
and  ESD  personnel  the  criteria  for  selection  of  the  18  ADP  systems  to 
be  surveyed.  The  criteria  were  those  stated  in  the  Phase  I  report; 

o  Selected  systems  must  be  stratified  by  size  (small,  medium, 

large)  and  by  functional  area  (similar  and  dissimilar) 

o  Selected  systems  must  have  undergone  a  fairly  recent  de¬ 
velopment  so  that  data  from  that  phase  will  still  be  available 

o  Selected  systems  must  not  present  any  unusual  security 

problems 

On  7  March  1966,  AFADA  selected  the  sample  of  18  ADP  systems;  but 
four  of  these  systems  subsequently  had  to  be  replaced  because  further 
investigation  revealed  an  extreme  scarcity  of  data  available  for  the  de¬ 
velopment  phase.  The  following  systems  were  replaced  during  data 
collection: 
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Original 


Replacement 


Tech  Order  Distribution  Data  Services  Workload  Control 

Tinker  AFB  Kelly  AFB 


Inventory  Management, 
Stock  Control,  Distribution 
W nght-Patter son  AFB 


Repair  Requirements  Computation 
System,  developed  at  W  right  - 
Patterson  AFB,  operated  at  Kelly  AFB 


IBM  505  Base  Supply 
Offut  AFB 


Base  Level  Inventory  Control  System 
Scott  AFB 


Engine  Management  System  MILSTAMP  Central  Data  Collection 

Tinker  AFB  System 

McClellan  AFB 


A  table  of  the  ADP  systems  in  the  final  sample  is  given  in  Table  2.  The 
orientation  of  approach  for  management  supporting  systems  (e.g.,  Base 
Supply  System)  and  operations  supporting  systems  (e.g.,  SPACETRACK) 
was  toward  "single  application"  as  objects  of  interest.  A  single  applica¬ 
tion  is  a  set  of  programs  dedicated  to  one  function  which  operates  on 
part  or  all  of  a  hardware  configuration.  The  research  and  development 
supporting  systems  in  the  sample  are  all  "scientific  job  shops"  where 
numerous  single  applications  exist  on  the  same  machine.  The  approach 
at  R<^  D  installations  was  to  select  one  of  the  many  single  applications  as 
an  object  of  interest. 


F.  Data  Collection  Procedures 


1 .  Initial  Design  of  Questionnaire 

Initial  design  of  the  questionnaire  to  be  used  in  field  collec¬ 
tion  of  data  was  based  on  the  work  of  Phase  I.  Appendix  I  of  the  Phase  I 
final  report  provided  a  partially  designed  questionnaire,  which  Phase  II 
project  members  used  as  a  point  of  departure  upon  which  to  apply  modifi¬ 
cations.  Project  members  were  assigned  specific  areas  of  the  question¬ 
naire  according  to  their  specialties.  Thus,  an  individual  with  extensive 
experience  in  programming  was  assigned  portions  of  the  questionnaire 
relative  to  programming,  while  an  individual  with  extensive  experience 
in  operations  was  assigned  a  questionnaire  section  dealing  with  computer 
operations.  The  questions  were  then  brought  together  and  organized  to 
form  a  comprehensive  questionnaire. 

Some  variables  originally  postulated  in  Phase  I,  such  as  overhead 
cost  and  facilities  cost,  were  not  included  in  the  questionnaire.  The  rea¬ 
son  these  costs  were  left  out  was  that  the  effort  required  to  gather  this 
type  of  cost  data  and  place  it  on  a  uniform  basis  could  be  much  more 
profitably  spent  on  more  central  areas  of  the  ADPS. 

The  initial  questionnaire  was  based  on  the  ADPS  model  described 
in  subsection  II. B  and  was  directed  toward  the  following  individuals: 


TABLE  2  -  ADP  SYSTEMS  IN  SAMPLE 
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1.  Installation  Manager:  To  supply  general  organizational, 
functional,  and  historic  information  on  system  development, 
operation,  and  use. 

2.  Systems/Programming  Supervisor:  To  supply  information 
on  development  costs,  documentation,  support  software,  ap¬ 
plication  software,  workload  descriptors,  and  personnel 
data  for  analyst s /programme rs . 

3.  Operations  Supervisor:  To  supply  information  on  job  sched¬ 
uling,  computer  utilization,  hardware  and  facility  problems, 
and  personnel  data  for  operators. 

The  initial  questionnaire  was  highly  detailed  and  of  relatively  fixed 
format;  i.e,,  it  was  composed  of  numerous  multiple  choice  and  specifi¬ 
cally  directed  questions  and  fixed  tabular  forms  for  recording  such  data 
as  workload  descriptors, 

2.  Modification  of  Questionnaire 


To  verify  the  usefulness  of  the  initial  questionnaire,  a  pilot 
data  collection  trip  was  made  to  Randolph  AFB,  Texas.  The  subject  sys¬ 
tem  was  the  Personnel  Data  System  for  Officers  (PDSO-65),  which  op¬ 
erated  on  the  Burroughs  B5500  computer.  This  system  was  an  excellent 
choice  for  pilot  data  collection,  since  it  included  a  very  broad  spectrum 
of  capabilities  and  features,  such  as  large  direct  access  memory,  on¬ 
line  inquiry  capability,  and  multiprogramming.  The  breadth  of  this  sys¬ 
tem  ensured  that  a  questionnaire  which  could  handle  it  would  be  applica¬ 
ble  *o  a  wide  variety  of  systems. 

Although  a  wealth  of  data  was  available  on  PDSO-65,  difficulty  was 
encountered  in  placing  this  data  in  the  rigid  format  of  the  initial  ques¬ 
tionnaire,  During  the  Randolph  pilot  data  collection,  the  questionnaire 
was  modified  to  conform  to  the  availability  and  type  of  data  that  was  en¬ 
countered,  and  the  data  were  recorded  on  the  modified  questionnaire. 

The  availability  of  data  at  Randolph,  particularly  in  the  proposal  and 
workload  areas,  suggested  that  highly  reliable  data  would  be  available  in 
all  areas  specified  by  Phase  I.  (Experience  in  subsequent  data  collection 
revealed  that  availability  of  data  on  PDSO-65  was  very  high.) 

On  returning  from  Randolph  AFB,  project  personnel  developed  a 
more  general  questionnaire.  The  revised  questionnaire  assumed  the  for¬ 
mat  of  an  interviewer’s  guideline  together  with  a  number  of  tabular 
sheets  for  entering  fixed  information.  A  copy  of  the  revised  question¬ 
naire  is  included  in  Appendix  B. 

3.  Letter  of  Introduction 


On  30  March  1966,  a  letter  introducing  the  project  to  all  in¬ 
stallations  to  be  interviewed  was  signed  for  Hewitt  T.  Wheless,  Lt.,  General, 
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USAF,  Assistant  Vice  Chief  of  Staff.  This  letter  (see  Figure  3)  assisted 
significantly  in  the  data  collectors1  receiving  outstanding  cooperation 
from  the  installations  visited. 

4.  Data  Collection  Trips 

In  addition  to  the  pilot  data  collection  trip  of  14  to  21  March 
1966,  three  other  series  of  trips  have  been  made.  The  first  series  of 
trips  covered  the  period  11  to  22  April,  the  second  covered  the  period 
9  to  20  May,  and  the  last  series  was  in  July  1  966 .  Eachteamhad  2  weeks 
to  cover  two  systems,  except  for  one  team  during  the  second  series, 
which  spent  1  week  on  one  system.  This  staffing  was  found  to  be  ade¬ 
quate,  and,  in  a  number  of  cases,  the  requisite  data  were  collected 
ahead  of  schedule.  An  average  of  about  8  man-days  per  system  was  re¬ 
quired  on  site  to  collect  data. 

Between  1  and  2  weeks  in  advance  of  the  data  collection  trips,  the 
lead  data  collection  teammember  contacted  the  AFADA-designated  con¬ 
tact  to  inform  him  of  the  purpose  of  the  trip  and  the  type  of  data  to  be 
collected.  Arrangements  for  time  and  place  of  meeting  on  arrival  at  the 
installation  were  also  made.  On  arriving  at  an  installation,  project  per¬ 
sonnel  briefed  key  installation  personnel  on  the  goals  of  the  project  and 
on  the  types  of  data  to  be  collected.  PRC,  in  turn,  asked  for  a  brief 
orientation  defining  organizational  responsibility  and  general  system 
characte  ristic  s . 

The  general  order  of  data  collection  was  as  follows: 

Day  _ Data  Collection  Activity _ 

1st  Briefing,  organizational,  and  functional 

de  s  cription 

2nd  System  proposal,  personnel,  and  manpower  data 

3rd  Programming  and  workload  data 

4th  Operations  and  workload  data 

5th  Review  of  all  data  and  debriefing 

The  last  task  was  always  a  debriefing  for  the  key  installation  personnel 
to  inform  them  of  the  data  that  had  been  collected.  Appendix  I  provides 
a  list  of  personnel  contacted  during  each  of  the  data  collection  trips. 

G.  Data  Reduction 


1.  Data  Summarization 


On  returning  from  a  data  collection  trip,  team  members  pro¬ 
ceeded  to  reduce  raw  numeric  data  to  consistent  meaningful  quantities, 
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DEPARTMENT  OF  THE  AIR  FORCE 
Offic~  of  t*>~  Chief  of  Staff 
United  States  Air  Force 

WASHINGTON.  r*.C. 


ATTN  or:  AFCCS 


IWM.YTO 


macti  Collection  of  Information  on  Automatic  Data  Processing  Systems 


toi  See  Distribution 

1.  In  1964  the  Secretary  of  the  Air  Force  asked  for  a  study  of  the  best 
way  to  use  Air  Force  Automatic  Data  Processing  Systems  (ADPS)  experience 
in  judging  proposals  for  new  automation.  Two  competitive  contracts  were 
awarded  by  the  Electronic  Systems  Division  of  the  Air  Force  Systems  Command 
to  develop  approaches  to  solving  this  problem.  As  a  result  of  the  compe¬ 
tition,  Planning  Research  Corporation  (PRC)  was  awarded  a  contract  to 
collect  data  on  18  existing  automatic  data  processing  systems  to  test  their 
proposed  approach  and  to  compile  an  experience  compendium.  The  systems  to 

be  examined  fall  in  the  areas  of  Logistics,  Personnel/Finance  and  Accounting,  ' 
Command  and  Control  and  R&D  Support.  A  list  of  them  is  attached. 

2.  This  will  be  the  first  known  attempt  to  develop  general  broad-spectrum 
techniques  for  estimating  costs  and  development  times  for  complete  data 
processing  systems  at  the  proposal  stage.  If  successful,  the  Air  Force 
will  receive  significant  benefits  from  the  effort. 

3.  The  data  collection  phase  of  the  contract  will  extend  through  July  1966. 
Air  Force  and  PRC  management  personnel  will  contact  each  facility  on  the 
attached  list  in  advance  of  the  data  collection  operation  to  brief  on  the 
project,  answer  questions,  and  arrange  details  of  the  later  data  collection 
visit. 

4.  In  order  to  reduce  the  impact  on  your  operations  and  to  secure  greater 
uniformity  of  data,  all  data  collection  will  be  done  by  PRC  personnel.  In 
general  the  data  to  be  gathered  will  include : 

a.  A  general  description  of  the  automatic  data  processing  system  and 
the  organizations  involved  in  its  development  and  use. 


b.  The  development  schedule,  both  an  planned  and  as  realized 


c.  The  development  and  operating  cost  history 


FIGURE  3  -  LETTER  OF  INTRODUCTION 
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which  had  been  identified  as  the  independent  and  dependent  variables  for 
statistical  analysis.  Examples  of  the  numeric  data  reduction  include 
computation  of  personnel  experience  averages,  percentages  of  computer 
hours  for  different  processing  functions,  and  total  characters/month  of 
input  volume. 

A  comprehensive  system  writeup  was  then  prepared.  This  docu¬ 
ment,  which  ranged  in  length  from  30  to  60  pages,  presented  a  narrative 
and  graphic  description  of  organizational  relationships,  system  history, 
proposal  for  the  system,  schedule,  system  design,  documentation,  pro¬ 
gramming,  file  conversion,  operations,  computer  utilization,  personnel, 
manpower,  and  future  plans.  The  system  writeup,  together  with  the  orig¬ 
inal  file  of  raw  data,  serves  as  the  basic  source  document  for  statistical 
analysis  inputs,  for  the  system  summaries  of  the  midpoint  report,  and 
for  the  system  descriptions  of  the  experience  handbook. 

The  numeric  and  narrative  data  reduction  required  about  12  man- 
days  per  system,  on  the  average. 

The  original  plan  for  Phase  II  called  for  a  file  maintenance  program 
to  be  developed  for  maintaining  and  reducing  the  raw  data  collected.  This 
effort  was  declared  unnecessary  after  the  pilot  data  collection  trip,  when 
it  appeared  that  raw  data  would  not  be  uniform  from  system  to  system. 

Summary  sheets  (see  Appendix  B)  for  the  numerical  data  collected 
were  prepared,  and  listed  all  dependent  and  independent  variables.  Var¬ 
iables  for  which  data  must  be  collected  were  indicated. 

The  data  from  the  summary  sheets  of  the  18  systems  were  com¬ 
piled  onto  work  sheets.  Copies  of  the  worksheets  were  provided  each 
data  collector  for  audit  and  recheck  of  the  data  he  was  responsible  for 
collecting.  Appendix  D  displays  the  worksheet  and  the  raw  data  collected 
for  each  of  the  18  systems.  An  indication  of  the  reliability  of  the  col¬ 
lected  data  is  also  given.  An  illustration  of  the  total  data  summariza¬ 
tion  process  is  depicted  in  Figure  4. 

2 .  Quality  Control  of  Data 


The  previous  discussion  of  operational  definitions  (subsection 
II. D)  described  the  redefinition  of  variables  following  the  second  and  third 
data  collection  trips.  In  addition  to  audit  and  recheck  of  data,  another  re¬ 
view  session  was  held  with  the  data  collectors  of  each  system.  The  pur¬ 
pose  was  to  evaluate  system  data  against  the  refined  definitions.  System 
data  was  assessed  for  accuracy  and  reliability,  completeness,  level  of 
detail  and  precision,  proper  categorizations,  and  currency. 

As  a  result  of  this  review,  some  variables  were  dropped  from  the 
data  analysis  because  of  too  many  missing  values.  Examples  are  aver¬ 
age  frequency  per  hour  of  input  and  hours  of  compilation,  assembly, 
checkout,  and  system  test  during  the  development  phase.  Because  of 
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FIGURE  4  -  DATA  REDUCTION 


the  small  sample  size,  the  normal  procedure  of  applying  the  mean  of 
the  variable  for  missing  values  may  distort  the  results  if  too  many  (for 
example,  more  than  two)  missing  values  were  substituted  in  this  way. 
Furthermore,  since  there  was  a  lack  of  "direct"  input  and  output  data 
(two  systems  had  such  data),  the  data  for  the  variable  were  merged 
with  batched  inputs  and  outputs.  "Direct"  refers  to  on-line  computer 
input  and  output  without  computer  operator  intervention.  (See  Appen¬ 
dix  C  for  definitions  of  input  and  output  variables  ,  *  .  .  ,  X^  . 

Other  variables  with  data  considered  unreliable  were  also  dropped 
from  the  analysis. 

H.  Findings 

Workload  and  cost  data  for  an  ADPS  were  generally  collectible  and 
reducible,  but  reliability  was  not  as  high  as  if  data  were  recorded  at  the 
time  of  event  occurrence.  The  problems  encountered  were  that  work¬ 
load  aid  cost  data  often  were  not  recorded  or  they  were  aggregated  such 
that  they  had  become  inseparable. 

Current  ADPS  proposals  do  not  contain  sufficient  data  about 
planned  costs  and  contain  nothing  about  workload  descriptors.  No  pro¬ 
posals  could  be  found  for  some  of  the  older  systems. 
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III.  DATA  ANALYSIS 


This  section  discusses  the  objectives  of,  the  procedures  followed 
for,  and  the  findings  of  the  data  analysis. 

A.  Objectives 


The  Phase  II  goals  for  data  analysis  were  as  follows: 

o  To  determine  relationships  between  ADPS  workload  de¬ 
scriptors  and  cost  and  development  time 

o  To  validate  that  the  workload  descriptors  exhibit  inclusive¬ 

ness  (intuitively  similar  systems  have  similar  workloads), 
exclusiveness  (intuitively  dissimilar  systems  have  signifi¬ 
cantly  different  workloads),  and  breadth  of  application 
(workload  descriptors  are  applicable  to  a  wide  variety  of 
system  s) 

o  To  refine  workload  descriptors  by  defining  them  more  pre¬ 
cisely,  by  eliminating  nonsignificant  descriptors,  and  by 
combining  significant  descriptors 

To  achieve  these  objectives  the  following  tasks  were  performed: 

o  Refining  the  workload  model  using  scatterplot  analysis  and 
correlation  analysis 

o  Testing  the  model  using  analysis  of  variance  and  analysis 
of  covariance 

o  Developing  cost  estimation  equations  using  multiple  regres¬ 
sion  analysis 

o  Determining  measures  of  reliability  for  the  cost  estimation 
equations 

o  Using  factor  analysis  to  discover  other  potential  relation¬ 
ships  and  to  check  the  cost  estimating  relationships  that 
were  derived 

Each  of  these  tasks  is  described  in  the  subsequent  discussion. 

Finally,  a  section  covering  findings  and  conclusions  completes  this  sec¬ 
tion  on  data  analysis. 

B.  Workload  Model  Refinement 


Following  data  reduction,  the  workoad  model  as  described  in  Sec¬ 
tion  II,  Data  Collection,  was  reduced  from  87  to  43  variables.  These 


21 


consisted  of  26  independent  variables,  constituting  the  preliminary  set 
of  estimating  factors,  and  17  dependent  variables,  constituting  the  pre¬ 
liminary  set  of  planning  factors.  These  variables  were  carefully  ex¬ 
amined  and  classified  into  logical  categories,  as  shown  in  Table  3  and 
defined  in  Appendix  C.  The  independent  variables  are  postulated  as 
the  causal  factors  and  the  dependent  variables  as  the  effects  on  the  cost 
and  other  phenomena  of  ADP  systems. 

1.  Selection  of  Factors  To  Be  Estimated 


The  initial  step  in  the  refinement  process  was  to  determine 
the  relevant  planning  factors  in  the  model.  These  are  the  factors  to  be 
estimated.  The  dependent  variables  were  critically  scrutinized  to  se¬ 
lect  those  factors  that  would  be  of  value  in  the  proposal  judging  process. 
The  single  criterion  employed  was  that  the  variables  chosen  should  con¬ 
sist  of  the  minimum  set  that  would  include  all  development  costs,  all 
operations  costs,  and  development  time.  The  following  factors  were 
selected: 

Cost  Factors 

Development  cost  variables 

Development  effort 
Program  checkout,  hardware  cost 
Operations  cost  variables 

Y^  Program  maintenance  personnel 
Y^  Operations  personnel 

Yj.  Application  production,  hardware  cost 
Y^  Program  maintenance,  hardware  cost 

Other  Factors 

Y^  Elapsed  development  time 

The  factor  Yz  (program  checkout,  hardware  cost)  had  to  be  elim¬ 
inated  from  the  statistical  analysis  because  of  insufficient  data;  many 
ADP  systems  did  not  record  this  information  separately  during  development. 
Source  statements  and  object  instructions,  although  interesting,  are 
not  included  because  it  would  be  preferable  to  obtain  development  cost 
directly.  Application  production  factors  do  not  appear  to  be  of  signifi¬ 
cant  import  for  proposal  evaluation.  As  mentioned  in  subsection  II. F.  1 , 
facilities  costs  were  not  included  in  the  study.  In  practice,  these  costs 
may  be  estimated  by  the  computer  size  and  the  number  of  personnel. 

2.  Selection  of  Factors  To  Be  Used  as  Predictors 


The  next  step  in  the  refinement  process  was  a  determina¬ 
tion  of  relevant  predictors  available  in  the  model.  These  predictors 
will  be  used  in  estimating  the  relevant  planning  factors.  The  independent 


22 


X  X 

4->  4-> 


G 

G 

M 

u 

Jh 

u 

Jh 

Jh 

Jh 

Jh 

Jh 

O 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

£ 

X 

X 

X 

X 

X 

X 

X 

X 

X 

0 

g 

E 

E 

E 

£ 

£ 

E 

E 

E 

E 

co 

0 

CO 

CO 

CO 

G 

G 

G 

£ 

G 

G 

G 

G 

Jh 

fn 

^4 

G 

G 

G 

G 

G 

G 

G 

G 

G 

£ 

G 

0 

4J 

(h 

(h 

4-J 

0 

4J 

(h 

0 

4-> 

(h 

4-J 

4-J 

4-J 

4-J 

4J 

4-J 

4J 

4-J 

0 

CO 

0  CO 

0 

CO 

0 

CO 

0  CO 

0 

CO 

0 

CO 

0 

CO 

0 

CO 

CO 

0 

0 

0 

G 

0 

0 

0 

0 

G 

G 

G 

G 

G 

G 

c 

G 

CO 

u 

CO  f-4 

CO 

Jh 

CO 

Jh 

CO  Jh 

CO 

J4 

CO  Jh 

CO 

J4 

CO 

Jh 

rt 

rt 

x 

x 

0 

rt 

x 

rt 

x 

0 

0 

0 

0 

0 

0 

0 

0 

rt 

d 

rt  rt 

rt 

rt 

rt 

rt 

rt  rt 

rt 

rt 

rt 

rt 

rt 

rt 

rt 

rt 

0 

fn 

e 

G 

0 

fn 

G 

fn 

G 

0 

0 

0 

0 

0 

0 

0 

0 

u 

0 

M  0 

u 

0 

Jh 

0 

Jh  0 

Jh 

0 

J4 

0 

Jh 

0 

Jh 

0 

2 

rt 

E 

E 

Jh 

rt 

E 

rt 

E 

fn 

fn 

U 

U 

u 

U 

M 

^4 

0 

0  >. 

0 

0 

>N 

0  >n 

0 

>N 

0 

>> 

0 

>> 

0 

>> 

X 

G 

G 

0 

X 

G 

X 

G 

0 

0 

0 

0 

0 

0 

0 

0 

> 

*4-4 

^  <4-t 

> 

VH 

> 

> 

v-t 

> 

VH 

> 

VH 

> 

»4H 

U 

X 

X 

cx, 

U 

z 

u 

£ 

PU, 

X 

X 

X 

X 

PU. 

X 

X 

< 

0 

<  0 

< 

0 

c 

0 

<  0 

< 

0 

< 

0 

< 

0 

< 

0 

CO 

X 

4-> 

g 

o 

2 


u 

o 

4-1 

u 

rt 

Uh 

co| 

.S 

4-> 

rt 

s 


w 


X 

d 

•  H 

G 

rt 

> 

4-» 

c 

0 

TJ 

C 

0 

a, 

0 

T3 

C 


o 

x 

s 

>S 

x 


U 


0 

E 

3 

fH 

o 

> 

4-> 

G 

a 

g 


CO 

0 

a 

>N 

4J 

o  ® 

•rH  TD 

4-»  i— I 

O  0) 

^  v-t 

CO 

G  rt 

rt  t! 

^  £ 

4->  Tj 

4->  4-> 

G 
a 
g 


G 

a 

g 


-h  rM  m 


X 

rt 

g 

rt 

> 

-M 

G 

a 

c 


0) 

a 

>> 

4-> 

T3 

G 

O 


<D 

U 

G 

G 

O 

CO 


0 

U 

G 

G 

o 

CO 


G 

0 

£ 

0 


C 

0) 

£ 

a; 

+-> 

rt 

-M 

CO 

0) 

u 


g 

0 

CUD 

rt 

G 

rt 

E 


to 

G 

0) 

CO 

rt 

G 

rt 

E 


g 

0 

CO 

rt 

G 

rt 


X 

aS 

G 

rt 

> 

4-1 

G 

a 


x 

d 

g 

d 

> 


d 

X 


d 

Q 


CD 

a. 

£ 

o 

U 


u 

o 

4-> 

u 

rt 

k 

0) 

u 

c 

0 

G 

0 

a| 

x 

w 

xi 
G 
rt 

G 

O 

•  H 
+-> 
aj 
V 
G 
X 

w 


XI 

d 

u 

d 

> 

G 

O 

4-> 

rt 

U 

G 

X 

o 

0 

CO 

0 

fH 

fH 

o 

U 


CO 

.3 

CO 

CO 

0 

u  ® 

O  »— H 

M  X 

a  rt 

•r-< 

rt  g 

M  rt 

45  > 

X 

0 

«  W 
•m  c 
rt  0 

E  ‘E 

O  0 

ti  a 
<  0 


0 

U 

G 

0 

G 

0 

a 

X 

0 

rt 

0 

g 

d 

t— H 

rt 

gJS 
3  -3 

O  rt 

3  * 

3  rt 
f*H  > 


CO 

4-J 

1 

d 

£ 

0 

Jh 

0 

CO 

ro 

G 

0 

Jh 

0 

VJ 

Jh 

G 

0 

CJ 

0 

CO 

o 

J4 

M 

G 

O 

4J 

G 

4-J 

G 

Jh 

0 

VL t 

a 

4-J 

G 

Jh 

0 

sy 

a 

V 

0 

*T» 

0 

0 

> 

Jh 

Jh 

0 

VH 

0 

CO 

cti 

0 

CO 

rt 

CO 

£  G 
TJ  0 

4_)  CO 

G 

•rH  G 

rt  0 

G  co 

0  £ 
“g 

0 

CO 

Jh 

G 

O 

0 

4-1 

G 

O 

co 

CO 

i— H 

0 

E 

a 

0 

E 

(X 

CO 

4-» 

£ 

£ 

rt 

Jh 

CO 

G 

O 

•  rH 

0 

E 

(X 

CO 

4-J 

E 

E 

rt 

J4 

CO 

G 

O 

•rH 

Jh 

4-J 

-M 

rt 

X 

X 

®  E 

£  £ 

U  £ 

0 

CO 

G 

CX 

0 

Li 

0 

r-H 

0 

r—i 

CO 

>N 

4-> 

rt 

0 

^H 

CO 

>N 

4J 

rt 

4-» 

CL 

p, 

rt 

rt 

-  « 

0  0 

CO 

^  p 

(-1 

Jh 

4-J 

0 

0 

1— 1 

CO 

Jh 

0 

i—4 

CO 

Jh 

G 

4-J 

4-1 

-M 

-M 

G  4-J 

0  4-J 

a  tj 

Jh 

L, 

E 

0 

G 

> 

> 

rt 

0 

0 

> 

rt 

0 

0 

p4 

G 

G 

rt 

rt 

fX  rt 

3  rt 

0  is 

0 

Q 

o 

G 

O 

0 

0 

G 

Jh 

a 

0 

G 

Jh 

a 

G 

t— i 

O 

o 

Q 

Q 

£\  m 

Ph  w 

« 

2 

CO 

U 

a 

a 

Q 

Q 

< 

Oh 

o 

Q 

< 

PU 

O 

o 

^H 

rvj 

CO 

X 

r- 

X 

o 

O 

^H 

fM 

X 

X 

IT) 

X 

X 

O 

^H 

^H 

pH 

^H 

r-H 

^H 

^H 

^H 

^H 

pH 

rM 

rM 

rM 

rM 

rM 

rM 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

G 

G 

4-» 

rt 

£ 

0 

G 

•rH 

X 

0 

rt 

2 


rg 

X 


C/) 

W 

J 

CO 

c 

2 

<: 

> 

g 

o 

2 

O 

H-t 

H 

<j 

u 


tf) 

tf) 

U 


CO 

W 

CP 

< 

H 


G 

O 

4-> 

O 

rt 

C*H 

co 

G 


X 

rt 

g 

rt 

> 

4-> 

G 

0 

X 

G 

0 

a« 

0 

Q 


>> 

CO 


G 

O 

£ 

i 

c 

rt 

2 


c 

0 

E 

a 

o 

0 

> 

0 

Q 


u 

d 

fH 

fH 

o 

Q 


G  _ 
0 


0 

X 

0 


E  £ 

rt  rt 
M  £ 

?-g 

PU  X 


M 

0 

X 

E 

G 


0 

0 

G 

rt 

G 

0 

■*-> 

c 

d 

£ 

E ' 

rt 

CO 

o 

u 

PU 


Jh 

E  ™ 

G  « 

O 

a  0 

.2  0 

4-1 

U 

5  3 

3  3 

rt 

0  d 

Jh  rt 

u* 

>  Jh 

0  3 

4-1 

CO 

«  a 

Q  > 

O  > 

O 

u 

u 

0 

X 

£ 

G 

£ 


0 

G 

G 

O 

co 

u 

0 

a 


G 

o 

• 

4-* 

rt 

0 

a 

O 


rt  4— > 
^  G 

O  o 
Q  £ 


0 

G 

X 

O 

U 


^  d 
G  ^ 
O  X 

J. 

-M  2 
rt  3 
0  ^ 
M  »v 

a  c 
a  o 

<  S 


^  LO 
>-t  >4 


^  X 

rt  Cl 

X  C 

0  2 
q  £ 


0 

0 

G 

rt 

G  4-> 

•rH  (n 

rt  o 
£  u 
c  ® 

E  m 

rt  rt 

CO^ 

2  M 
rt 

P^  X 


X 


u 

0 

X 

£ 

G 

X 


U 

0 

X 

£ 

G 

X 


4-J 

G 

4-J 

c 

4-J 

G 

4-» 

c 

4-* 

c 

4-J 

c 

4-J 

G 

-M 

G 

0 

0 

0 

0 

0 

0 

0 

0 

u 

u 

a 

u 

u 

u 

U 

U 

Jh 

Jh 

Jh 

Jh 

Jh 

Jh 

Jh 

Jh 

0 

0 

0 

0 

0 

0 

0 

0 

PU 

Oh 

PU 

Dh 

Oh 

PU 

Oh 

PU 

G 

0 

£ 

a 

co 

4-> 

G 

CO 

G 

O 

co 

Jh 

0 

V 

rs 

G 

O 

r  s 

G 

O 

4-> 

o 

rC 

G 

0 

4-J 

G 

0 

0 

> 

0 

E 

0 

4-* 

4-J 

U 

G 

Jh 

G 

O 

X 

c 

d 

G 

0 

G 

O 

X 

4-J 

rt 

Jh 

0 

G 

O 

X 

u 

G 

TJ 

O 

O 

4-J 

u 

G 

r O 
O 
Jh 

a 

0 

rt 

CO 

G 

4-> 

G 

G 

G 

Jh 

T3 

T3 

TJ 

-M 

CO 

G 

4-J 
•  rH 

T3 

O 

•rH 

4-* 

G 

•  fH 

rt 

O 

•  rH 
+-> 

0 

CO 

O 

•  fH 
+-> 

a 

O 

Jh 

0* 

4-J 

0 

0 

4-* 

0 

u 

C 

u 

4-J 

u 

0 

CX 

G 

CO 

u 

U 

G 

G 

G 

Jh 

G 

X 

CO 

CX 

CO 

a 

0 

Jh 

0 

G 

TJ 

0 

"O 

0 

T3 

Jh 

Jh 

4-J 

c 

G 

rt 

G 

G 

Cl 

O 

r-H 

0 

(X 

O 

0 

G 

Jh 

G 

G 

f-H 

W 

G 

-  rH 

O 

X 

X 

O 

G 

i— i 

Jh 

a 

ix 

Jh 

a 

0 

oi 

Jh 

Oh 

2 

O 

X 

O 

X 

O 

u 

O 

X 

O  — 

ao  (7s  — i  — 

>4  >4  >4  >4 


G 

O 

4— > 

0 

G 

o 

PU 

G 

O 

• 

rt 

0 

a 

a 


T^l  LO 
>4  >4 


x  r— 

>4  >4 


CO 

O 

0 

0 

U 

d 

Z 

X 

u 

d 

X 

co 

G 

4-1 

G 

a 

E 

o 

0 

U-4 

o 

X 

O 

X 

4-> 

0 

£ 

u 

o 


W 

X 

•  fH 

X 

G 

0 

a 

a 

< 

0 

0 

CO 


0 

4-1 

o 


variables  were  critically  scrutinized  to  select  those  factors  that  are 
causally  related  to  the  planning  factors  and  that  can  be  available  at  pro¬ 
posal  evaluation  time  from  the  ADPS  proposal.  The  latter  criterion 
was  considered  to  be  of  paramount  importance  because  estimating  rela¬ 
tionships  that  were  derived  from  nonavailable  factors  cannot  be  put  to 
use.  The  following  factors  were  selected: 

Workload  Descriptors 

Input  variables 

Input  volume 
Input  transaction  types 

X^  Input  data  fields 

Output  variables 

X^  Output  volume 

X^  Output  formats 

Data  base  variables 

X^  Data  base  (size) 

X^  Data  base  record  types 

The  factor  X4  (input  rejects)  was  not  included  because  it  may  not  be  de¬ 
terminable  at  proposal  preparation  time.  While  the  complexity  factor, 
education  and  experience  factor,  and  machine  maturity  factor  are  im¬ 
portant,  they  are  not  usually  known  at  the  proposal  preparation  time. 
Complexity  is  difficult  to  determine  until  system  design  is  nearly  com¬ 
plete;  the  machine  (computer)  to  be  used  is  usually  acquired  subse¬ 
quent  to  proposal  approval;  and  education  and  experience  levels  of  per¬ 
sonnel  are  largely  unknown  until  the  proposal  is  being  implemented  and 
staffing  largely  completed. 

3.  General  Linear  Model 


The  foundation  for  the  use  of  regression  analysis  is  estab- 
listed  in  this  subsection  and  in  the  one  immediately  following.  These 
subsections  do  not  pertain  specifically  to  the  subject  at  hand  and  may 
be  omitted  by  readers  who  are  not  interested  in  statistical  methods. 

It  is  assumed  that  a  linear  stochastic  relationship  exists  between 
the  dependent  variables  Y.  (cost  variables)  and  a  set  of  independent 
variables  X,  ,  X-.,  .  .  .  ,  X ^  (workload  descriptors)  and  that  this  rela- 
tionship  may  be  written  as 


Y.  =  a.  + 

J  J 


m 

2 

i=l 


S..X.  +  u. 
1J  1  J 


where  j  =  1 ,  2,  •  •  ,  n  defines  the  set  of  cost  variables,  and  the  fre¬ 

quency  distribution  of  error  terms  aj  ,  (3lj,  (32j,  •••,  (3m j,  and  f(Uj), 
define  the  data  generating  mechanism.  The  problem  is  to  specify  the 
dependency  relationship  correctly  by  selecting  a  proper  set  of  independ¬ 
ent  variables,  a  proper  functional  form,  and  a  vector  of  parameters 
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aj,  t>lj>  b .  .  .  ,  bmj  that  provides  a  good  set  of  estimates  for  aj, 

Plj>  •  •  •  *  Pmj>  tne  structural  coefficients  of  the  underlying 

population  (see  Reference  10). 

The  estimates  a-^,  blj,  b2j ,  •  •  •  ,  bmj  are  obtained  by  regression 
analysis,  and  Uj '  s  are  determined  and  evaluated  by  measures  of  relia¬ 
bility.  Both  topics  will  be  covered  subsequently  in  this  section. 

4.  Requirements  for  Estimation  Efficiency 

Estimation  efficiency  means  the  accuracy  of  the  estimating 
relationship.  The  estimation  efficiency  of  the  prediction  equation  de¬ 
rived  by  means  of  regression  analysis  is  conditioned  on  the  following 
requirements; 

1.  E(Yj)  -  a j  +  £bijX^  is  linear  in  the  specified  set  of  param¬ 
eters  and  independent  variables. 

2.  f(Uj)  normal;  the  conditional  distribution  of  Yl  given 
Xi ,  X2,  -  •  *  >  xm  follows  the  normal  probability  function. 

3.  E(UjUj-fs)  -  0  for  all  S  4  a  ;  successive  errors  are  inde¬ 
pendently  distributed. 

4.  E(Uj^)  is  constant  (the  variance  of  error  terms  is  independ¬ 
ent  of  the  size  of  explanatory  variables  X^);  absence  of 
hete roscedasticity  (heteroscedasticity  refers  to  the  nonuni¬ 
formity  of  the  variance  of  the  Y  variable  through  the  range 
of  the  X  variable). 

5.  E(XiXk)  =  0;  independent  variables  are  independent  of  one 
another  (absence  of  multicollinearity). 

6.  E(UjX1)  =  0,  for  all  i  =  1,  2,  .  .  .  ,  m;  the  requirement  that 

Y  be  dependent  on  X  but  not  vice-versa  (absence  of  feedback). 

These  requirements  were  checked  by  the  use  of  scatterplot  analysis  and 
correlation  analysis,  which  will  be  covered  in  subsequent  subsections. 

The  residuals  appeared  normal  when  plotted  as  transformed. 

5 .  Scatterplot  Analysis 

The  purpose  of  the  scatterplot  was  to  provide  for  visual  in¬ 
spection  of  the  distribution  of  variables.  (See  subsection  B  of  Appendix 
E  for  the  methodology  used  for  generating  scatterplots.)  An  examination 
of  the  plots  of  independent  variables  against  the  dependent  variables 
showed  tight  clustering  of  data  points  with  an  extreme  outlier  or  a  fan¬ 
ning  out  of  data  at  the  higher  ends  of  each  scale  (see  Figure  5).  This 
indicated  a  lack  of  linearity  of  the  Y  and  X  relationship  and  nonuniformity 
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of  the  Y  variable  through  the  range  of  the  X  variable,  which  violate 
requirements  for  the  use  of  regression  analysis. 

In  attempting  to  improve  the  distribution  of  data,  transformation 
of  variables  was  performed.  Logarithmic  (base  10)  transformation  of 
the  independent  variable  only,  log^Q  transformation  of  the  dependent 
variable  only,  and  logjQ  transformation  of  both  variables  were  tried. 

It  was  found  that  log^Q  transformation  of  both  variables  was  necessary 
to  produce  a  resulting  distribution  that  appeared  rectilinear  and  suitable 
for  analysis.  Figure  6  shows  the  results  of  log  10  transformation  on 
the  same  variables  shown  in  Figure  5.  (See  subsection  C  of  Appendix  E 
for  the  methodology  used  for  transformation  of  variables.) 

Transformation  techniques  are  intended  to  manipulate  the  data  so 
that  the  resulting  distribution  of  data  will  match  the  assumptions  de¬ 
manded  for  using  linear  regression  analysis.  The  use  of  transforma¬ 
tion  is  not  intended  to  improve  the  results  nor  the  reliability  of  the  es¬ 
timation  equations  derived.  Furthermore,  while  all  computations  to 
determine  reliability  are  performed  in  the  transformed  state,  the  ulti¬ 
mate  estimates  must  be  retransformed  before  use. 


Figures  6  through  10  display  examples  of  scatte rplot s .  These 
show  the  good  correlations  between  the  primary  workload  descriptor 
against  each  of  the  five  cost  variables.  (See  also  Table  4.) 


6 .  Correlation  Analysis 

A  correlation  matrix  for  all  transformed  independent  and 
dependent  variables  was  computed.  (See  subsection  D  in  Appendix  E 
for  methodology  used  for  correlation.)  This  matrix  was  closely  exam¬ 
ined,  and  each  cost  variable  Yj  is  postulated  to  be  a  function  of  all 
workload  descriptors  that  are  significantly  correlated  with  Yj  for 
that  given  sample  size. 


The  causal  relationships  between  each  Yj  and  remaining  X^’s 
were  analyzed  to  ensure  that  the  effect  of  Yj  is  caused  by  the  X^’s  but 
that  Yj  has  no  cause-effect  relationships  with  the  Xi’s.  None  of  the 
latter  relationship  was  found. 


Finally,  the  X^'s  for  each  Y ;  were  examined  for  intercorrela¬ 
tions.  When  significant  intercorrelations  existed,  one  or  more  vari¬ 
ables  were  deleted  until  there  was  an  absence  of  intercor related  vari¬ 
ables.  The  results  of  the  correlation  analysis  are  summarized  in  the 
cost  model  shown  in  Table  4.  The  regression  model  (the  model  enter¬ 
ing  the  regression  analysis)  contained  only  the  variables  remaining 
following  the  elimination  process  previously  described.  The  planning 
factor,  elapsed  development  time  (Yy),  was  dropped  because  of  lack  of 
significant  correlation  with  any  workload  descriptor.  This  was  prob¬ 
ably  because  of  small  sample  size  that  did  not  supply  sufficient  data 
points  to  establish  this  relationship.  Elapsed  development  time  may 
still  be  estimated  as  a  function  of  the  number  of  man-months  of  devel¬ 
opment  effort. 
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FIGURE  5  -  MAN-MONTHS  OF  DEVELOPMENT  EFFORT  VERSUS  NUMBER  OF  INPUT  DATA  FIELDS 
(BEFORE  LOG,  n  TRANSFORMATION) 
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FIGURE  6  -  MAN-MONTHS  OF  DEVELOPMENT  EFFORT  VERSUS  NUMBER  OF  INPUT  DATA  FIELDS 
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FIGURE  7  -  NUMBER  OF  PROGRAM  MAINTENANCE  PERSONNEL  VERSUS  NUMBER  OF  INPUT 
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FIGURE  8  -  NUMBER  OF  OPERATIONS  PERSONNEL  VERSUS  NUMBER  OF  OUTPUT  FORMATS 
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FIGURE  9  -  DOLLARS  PER  MONTH  OF  HARDWARE  COST  FOR  APPLICATION  PRODUCTION 
VERSUS  NUMBER  OF  OUTPUT  FORMATS 
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FIGURE  10  -  DOLLARS  PER  MONTH  OF  HARDWARE  COST  FOR  PROGRAM  MAINTENANCE 
VERSUS  NUMBER  OF  OUTPUT  FORMATS 
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F  air 


C.  Testing  the  Model 

The  ADPS  sample  was  stratified  into  four  sub  samples  (  see  Table  Z) . 
These  were  (1)  management  supporting  data  systems  - -logistic s ;  (Z)  man¬ 
agement  supporting  data  systems  - -per sonnel /finance ,  (3)  operations 

supporting  data  systems,  and  (4)  research  and  development  supporting 
systems.  The  subsamples  were  tested  to  determine  whether  their  cost 
variables  belong  in  the  same  population  or  should  be  treated  as  inde¬ 
pendent.  Three  tests  were  employed: 

1.  t-test  to  determine  whether  each  of  two  subsamples  belong 
in  the  same  population. 

Z.  One-way  analysis  of  variance  to  determine  whether  all  four 
subsamples  belong  in  the  same  population. 

3.  Analysis  of  covariance  to  determine,  after  adjusting  for  dif¬ 
ferences  in  the  workload  descriptors,  whether  all  four  sub¬ 
samples  belong  in  the  same  population. 

These  tests  were  conducted  for  all  five  cost  variables;  the  results  are 
summarized  in  Table  5.  (See  subsection  E  in  Appendix  E  for  method¬ 
ology  used  for  analysis  of  variance.)  The  results  were  mixed  and  the 
18  systems  were  treated  as  members  of  the  same  population. 

Because  relationships  exist  between  workload  descriptors  and 
cost  variables  as  demonstrated  by  the  regression  model,  it  should  fol¬ 
low  that,  if  the  preceding  tests  were  all  significant,  then  the  hypothesis 
that  workload  descriptors  exhibit  inclusiveness  and  exclusivness  would 
be  validated.  The  general  applicability  of  workload  descriptors  has 
proved  that  they  exhibit  breadth  of  application.  However,  the  results 
of  the  tests  were  mixed,  and  the  hypothesis  was  not  validated.  One¬ 
way  analysis  of  variance  was  also  performed  on  the  workload  descrip¬ 
tors.  The  results  were  also  mixed;  however,  these  results  do  not 
mean  that  workload  descriptors  do  not  exhibit  inclusiveness  and  exclu¬ 
siveness.  The  mixed  results  are  very  likely  due  to  the  extremely 
small  subsample  sizes. 


D.  Regression  Analysis 

Multiple  regression  analysis  using  the  stepwise  regression  pro¬ 
cedure  was  applied  to  the  cost  models  and  five  cost  estimation  equations 
were  derived.  Reliability  measures  (coefficients  of  variations  and  co¬ 
efficients  of  correlation)  and  equations  for  the  80  percent  prediction  in¬ 
tervals  were  computed  for  each  of  the  five  estimating  relationships. 

The  equations  are  summarized  in  Table  6.  (The  statistical  procedure 
is  described  in  subsection  F  of  Appendix  E.) 

For  each  cost  variable,  three  cost  estimates  are  obtained  by  using 
the  appropriate  workload  descriptors  and  then  by  solving  the  prediction 
(cost  estimation)  equation  and  the  prediction  interval  equation.  The 
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TABLE  5  -  RESULTS  OF  t-TESTS,  ONE-WAY  ANALYSIS  OF  VARIANCE,  AND  ANALYSIS  OF 
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(3)  0.01--Means  reject  hypothesis  at  a  =  0.01,  99  percent  level  of  confidence 

(4)  Subscripts  on  \j,  :  1 --Management  supporting  data  sy stems--logistics 

2- -Management  supporting  data  systems--per sonnel/finance 

3- -Research  and  development  supporting  systems 

4- -Operations  supporting  data  systems 


TABLE  6  -  COST  ESTIMATION  EQUATIONS 
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=  logjQ  (predicted  number  of  operations  personnel)  =  logjg  (characters  per  month  of  output  volume) 

logjO  (predicted  dollars  per  month  of  hardware  cost  for  application  production)  =  logjg  (number  of  output  formats) 

=  1  o  g  I  q  (predicted  dollars  per  month  of  hardware  cost  for  program  maintenance)  X_  log.  .  (characters  in  data  base) 


prediction  equation  gives  a  single-point  cost  estimate  that  may  be  in¬ 
terpreted  in  the  following  manner:  The  cost  is  expected  to  be  this  value; 
50  percent  of  the  time  it  is  expected  to  be  greater,  and  50  percent  of  the 
time  it  is  expected  to  be  less.  The  prediction  interval  provides  a  range 
estimate  of  cost.  The  solution  of  the  prediction  interval  equation  gives 
upper  and  lower  limits  that  bracket  the  expected  value  and  that  may  be 
interpreted  in  the  following  manner:  Cost  is  expected  to  be  less  than 
the  upper  limit  90  percent  of  the  time  and  greater  than  the  lower  limit 
90  percent  of  the  time. 

For  the  Experience  Handbook,  these  equations  will  be  pre -solved 
with  graphical  methods.  Figure  13  displays  three  iso-graphs  represent¬ 
ing  Equation  I. 

Multiple  r  eg  re  s  sion  analysis  was  also  applied  to  the  1  2  management¬ 
supporting  data  systems.  The  resulting  estimation  equations  for  the 
five  cost  factors  possessed  prediction  intervals  of  greater  magnitude 
than  the  equations  developed  for  all  18  systems.  Again,  the  smaller 
sample  size  was  a  major  contributor  to  the  larger  variance. 

The  estimating  equations  displayed  wide  prediction  intervals.  As 
an  example,  using  Equation  I  with  workload  descriptors  of  100  for  the 
number  of  input  data  fields  and  10  for  the  number  of  output  formats,  the 
solution  obtained  is  153  for  the  number  of  estimated  man-months  of  de¬ 
velopment  effort  with  a  prediction  interval  of  38  to  614  man-months. 

With  an  increase  in  sample  size  from  18  to  N,  the  interval  width  would 
decrease  by  a  factor  of  ^8/N  of  the  logio  values  of  the  interval;  if 
N  =  180  ,  the  interval  limits  will  be  99  to  238  man-months. 

The  regression  analyses  undertaken  all  assumed  a  linear  model 
with  logio  transformed  variables.  Because  of  the  small  sample  size, 
it  was  judged  that  polynominal  regression  would  not  provide  estimating 
equations  of  better  efficiency;  therefore,  it  was  not  used. 

E.  Factor  Analysis 


Investigations  leading  toward  development  of  a  cost  model  incor¬ 
porating  complexity  factors  and  personnel  factors  were  not  fruitful. 
Factor  analysis  was  used  in  an  attempt  to  discover  potential  significant 
relationships  between  these  variables.  (See  subsection  G  in  AppendixE 
for  the  methodology  used  in  factor  analysis.) 

Of  the  43  variables  in  the  original  reduced  workload  model,  31 
were  entered  into  the  factor  analysis;  12  were  not  entered  because  of 
insufficient  data  points.  Only  those  variables  with  17  or  18  observa¬ 
tions  were  retained  because  of  the  requirements  for  uniformity  in  sam¬ 
ple  size.  The  eliminated  variables  had  between  11  and  14  observations 
each,  with  one  having  16  observations.  The  following  variables  were 
not  entered: 
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Dependent  Variables 


10 


11 


12 


Hardware  cost  for  program  checkout 
Percent  of  production  hours  for  input  edit 
Percent  of  production  hours  for  file  maintenance 
Percent  of  production  hours  for  report  generation 


^13  Percent  of  production  hours  for  merge 


14 


Percent  of  production  hours  for  sort 


Percent  of  production  hours  for  compute 


16 


17 


Percent  of  production  hours  for  query 
Percent  of  production  hours  for  control 


Independent  Variables 


X 


17 


Years  of  college  education  for  development  managers 


X1Q  Years  of  ADP  experience  for  operations  personnel 
i  o 

X25  Years  of  functional  area  experience  for  operations 
personnel 

Eight  of  the  omitted  dependent  variables  (Yio  through  Y17)  were  not 
significant  with  respect  to  ultimate  utility  because  they  were  not  cost 
variables.  Therefore,  their  loss  did  not  detract  substantially  from  the 
analysis. 


The  intermediate  results  showed  17  nonzero  eigenvalues  and  sub¬ 
sequently  the  factor  matrix  was  rotated  17  times.  Interpretation  of  the 
results  of  this  sample  following  rotation  of  the  factor  matrix  did  not  in¬ 
dicate  that  significant  relationships  existed  between  complexity  variables 
and  cost  variables,  and  between  education  and  experience  variables  and 
cost  variables.  The  factor  analysis  did  indicate,  however,  that  signifi¬ 
cant  relationships  existed  between  the  workload  descriptors  and  cost 
factors  of  the  cost  estimating  equations.  This  result  was  consistent 
with  the  results  obtained  previously.  Factor  loadings  for  selected  var¬ 
iables  from  two  of  the  17  rotations  of  the  factor  matrix  are  displayed  in 
Table  7.  The  high  factor  loadings  are  underlined.  None  of  the  other  ro¬ 
tations  showed  any  significantly  high  factor  loadings  in  one  or  more  de¬ 
pendent  variables  with  one  or  more  independent  variables. 
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TABLE  7  -  FACTOR  LOADINGS  FOR  SELECTED  VARIABLES 


Independent  Variables 


Symbol 

Name 

F actor  I 

F actor  II 

xi 

Characters  per  month  of  input  volume 

0.73 

0.03 

X2 

Number  of  input  transaction  types 

0.18 

0.68 

X3 

Number  of  input  data  fields 

0.21 

0.70 

X5 

Characters  per  month  of  output  volume 

0.72 

0.02 

X6 

Number  of  output  formats 

0.68 

0.39 

X7 

Characters  in  data  base 

0.31 

0.73 

X8 

Number  of  data  base  record  types 

0.24 

0.85 

Dependent  Variables 

Symbol 

Name 

Factor  I 

F actor  II 

Yi 

Man-months  of  development  effort 

0.40 

0.63 

Y3 

Number  of  program  maintenance 
personnel 

0.51 

0.77 

Y4 

Number  of  operations  personnel 

0.74 

0.52 

Y5 

Dollars  per  month  of  hardware  cost 
for  application  production 

0.93 

0.22 

Y6 

Dollars  per  month  of  hardware  cost 
for  program  maintenance 

0.91 

0.17 

Note:  High  factor  loadings  are  underlined. 
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F.  Findings  and  Conclusions 


1  .  Findings 

Workload  descriptors  were  refined  by  defining  them  more 
precisely,  by  eliminating  nonsignificant  descriptors,  and  by  combining 
significant  descriptors.  The  small  set  of  workload  descriptors  con¬ 
sists  of  the  following: 

o  Characters  per  month  of  input  volume 

o  Number  of  input  data  fields 

o  Characters  per  month  of  output  volume 

o  Number  of  output  formats 

o  Characters  in  data  base 

Workload  descriptors  were  shown  to  have  breadth  of  application; 
that  is,  they  can  be  applied  to  a  wide  variety  of  systems.  Workload 
descriptors  were  not  conclusively  proven  to  exhibit  inclusiveness  and 
exclusiveness;  that  is,  they  are  similar  for  some  similar  systems  and 
dissimilar  for  some  dissimilar  systems,  but  not  always  consistently  so. 

Relationships  between  workload  descriptors  and  costs  were  de¬ 
termined,  and  cost  estimation  equations  were  derived  for  the  following 
cost  variables: 

o  Man-months  of  development  effort 

o  Number  of  program  maintenance  personnel 

o  Number  of  operations  personnel 

o  Dollars  per  month  of  hardware  cost  for  application 
production 

o  Dollars  per  month  of  hardware  cost  for  program 
maintenance 

The  relationship  between  workload  descriptors  and  elapsed  development 
time  were  not  derivable  from  the  18-ADPS  sample,  but  may  be  obtained 
as  a  function  of  the  number  of  man-months  of  development  effort. 

The  estimating  relationships  derived  from  the  18-ADPS  sample 
displayed  wide  prediction  intervals.  With  an  increase  in  sample  size 
from  18  to  N,  the  log^o  interval  width  would  be  decreased  at  least  by  a 
factor  of  y/\  8  /N  if  everything  else  remained  the  same  or  improved. 

2.  Conclusion 


Relationships  do  exist  between  workload  descriptors  and  costs. 
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IV.  EXPERIENCE  HANDBOOK  DEVELOPMENT 


This  section  discusses  the  objectives  of  and  the  activities  leading 
to  the  development  of  the  ADP  Experience  Handbook  (Pilot  Version). 

A.  Objectives 

The  Phase  II  goals  for  development  of  the  Experience  Handbook 
were  as  follows: 

°  Develop  a  method  to  facilitate  the  solution  of  cost  esti¬ 
mation  equations 

o  Write  18  AD  PS  macrodescriptions 

o  Develop  indexing  schemes  for  finding  portions  of  the 

macrodescriptions  relevant  to  a  proposed  AD  PS  based 
on  attributes  of  the  proposed  AD  PS 

o  Organize  solutions  to  cost  estimation  equations,  macro¬ 
descriptions,  and  indexes  into  a  usable  handbook 

To  achieve  these  objectives,  the  following  tasks  were  performed: 

o  Development  of  cost  estimation  graphs 

o  Development  of  f,totaln  system  descriptions  of  fixed 
format  consisting  of  seven  pages  of  highly  distilled 
information 

o  Development  of  indexing  methods  and  procedures  for 
discriminating  retrieval  of  relevant  data 

o  Organization  of  a  handbook  in  easily  usable  form 

o  Development  of  procedures  tor  use  of  the  handbook 

o  Establishment  of  a  glossary  of  terms 

o  Preparation  of  a  primer  as  a  more  detailed  example 

of  how  the  handbook  is  used 

Each  of  these  tasks  is  described  in  the  subsequent  sections,  followed  by 
a  section  on  findings  and  conclusions. 
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B. 


Cost  Estimation  Graphs 


1 .  Rationale  for  Development  of  Graphs 

An  important  step  in  judging  proposals  for  new  automation 
is  to  conduct  a  cost  analysis.  The  planned  costs  for  development  and 
operation  of  an  ADPS  would  be  compared  to  predicted  costs  that  were 
determined  by  use  of  cost  estimation  equations.  Phase  II  cost  estima¬ 
tion  equations  are  given  in  Table  6,  along  with  equations  for  determin¬ 
ing  their  80  percent  prediction  intervals. 

The  solution  of  these  equations  would  require  the  transformation 
of  the  X  variables  by  use  of  logarithms  and  finally  the  retransformation 
of  the  results  by  obtaining  the  antilogarithms.  In  addition,  a  square 
root  must  be  computed  during  the  computation  of  the  prediction  interval. 
The  user  of  these  equations  must  therefore  be  conversant  in  the  use  of 
logarithms.  The  calculation  itself  is  quite  a  burdensome  chore  subject 
to  clerical  errors.  As  a  result,  several  methods  were  investigated  to 
provide  means  for  aiding  the  proposal  evaluator  in  this  task.  Two  of 
these  methods  are  described  in  the  following  paragraphs. 

2.  Graphical  Aids  to  Computation 

a.  Nomographs 

A  set  of  nomographs  was  developed  to  aid  in  computing 
the  predicted  cost  values  and  the  prediction  intervals.  An  example  of 
the  nomographs  for  the  solution  of  one  set  of  equations  is  given  in  Fig¬ 
ures  11  and  12.  The  use  of  these  nomographs  to  obtain  the  predicted 
value  and  prediction  interval  would  entail  the  following  steps: 

o  Locating  six  points  on  the  scales 

o  Drawing  four  straight  lines 

o  Reading  two  values 

o  Manually  adding  and  subtracting  two  values 

o  Locating  three  additional  points  on  a  scale 

o  *  Reading  three  additional  values 

This  method  substantially  reduced  the  computational  effort  but  was  still 
rather  cumbersome.  Therefore,  the  search  for  simpler  methods  was 
continued. 


b.  Iso-Graphs 

A  set  of  iso-graphs  was  then  developed  in  an  attempt 
to  further  simplify  the  computation  task.  An  example  of  the  iso-graphs 
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FIGURE  11  -  PREDICTION  EQUATION  NOMOGRAPH  FOR  MAN-MONTHS  OF  DEVELOPMENT  EFFORT 
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Number  Of  Output  Formats  Number  Of  Output  format  Number  of  Output 


Co.t  Estimating  Procedure  for  Number  of  Program  Maintenance  Personnel 

1.  Find  the  vaiue  of  Number  of  Input  Data  Field.  for  the  proposed 
ADPS  on  the  horizontal  scale  of  any  one  of  the  three  lao-graphs. 

2.  Draw  a  vertical  ilne  through  all  three  lao-grapha  at  the  value 
e atabiiahed  In  Step  1. 

3.  Find  the  value  of  Number  of  Output  Formata  for  the  propoaed  ADPS 
on  the  vertical  acaie  of  each  of  the  three  lao-grapha. 

4.  Draw  a  horizontal  line  on  ail  three  lao-grapha  through  the  vaiuea 
eatabllahed  In  Step  3. 

5.  On  the  top  iao  graph,  determine  the  vaiue  that  Number  of  Program 
Maintenance  Personnel  is  expected  to  be  ieaa  than,  90  percent  of 
the  time,  by  logarithmically  interpolating  the  intersection  point 

of  the  vertical  (Step  2)  and  horizontal  (Step  4)  tinea  between  adjacent 
iao-iinea. 

6.  On  the  center  iao-gr.tph,  determine  the  vaiue  that  Number  of  Program 
Maintenance  Personnel  ia  expected  to  be,  by  logarithmically 
Interpolating  the  intersection  point  of  the  vertical  (Step  2J  and 
horizontal  (Step  4)  lines  between  adjacent  iao-linea. 

7.  On  the  bottom  iso-graph,  determine  the  value  that  Number  of  Program 
Maintenance  Personnel  ia  expected  to  be  greater  than, 90  percent 

of  the  time,  by  logarithmically  interpolating  the  intersection  point 
of  the  vertical  (Step  £)  and  horizontal  (Step  4)  lines  between  adjacent 
iao-iines . 


FIGURE  13  -  EXAMPLE  OF  COST  ESTIMATING  ISO-GRAPHS  FOR  NUMBER  OF  PROGRAM 
MAINTENANCE  PERSONNEL 


for  one  set  of  equations  is  given  in  Figure  13.  The  use  of  iso-graphs 
to  obtain  the  predicted  value  and  the  prediction  interval  would  entail: 

1.  Locating  four  points  on  the  scales. 

2.  Drawing  four  straight  lines. 

3.  Reading  three  values. 

This  method,  which  require s  considerably  less  effort  than  using  the  nomo¬ 
graphs,  is  the  method  selected  for  use  in  the  ADP  Experience  Handbook. 
Workload  descriptors  are  used  to  retrieve  the  cost  estimates.  Instruc¬ 
tions  for  using  the  iso-graphs  can  be  found  on  the  charts.  For  more 
detailed  descriptions  and  instructions  and  a  complete  set  of  these  iso¬ 
graphs,  see  the  ADP  Experience  Handbook. 

C.  System  Descriptions 

A  seven-page  system  description  was  developed  for  each  system, 
using  data  collected  in  this  phase.  The  descriptions,  which  are  included 
in  the  Air  Force  ADP  Experience  Handbook  (Pilot  Version),  are  highly 
formatted  and  standardized  to  provide  rapid  cognition  of  system  prob¬ 
lems  and  attributes  and  to  enhance  cross-system  comparisons.  Each 
system  description  presents  a  total  system  picture  which  comprises 
the  following  21  sections: 

o  System 

o  Data  System  Designator 

o  Data  Collection  Date 

o  Location 

o  Function 

o  Organization 

o  History 

o  Schedule 

o  Description 

o  Workload 

o  Hardware 

o  Software 

o  Application  Program  Development 
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O  File  Conversion 

o  Documentation 

o  Personnel 

o  Operations 

o  Application  Program  Maintenance 

o  Benefits 

o  Cost  Factors 

o  Future  Plans 

Information  from  the  system  descriptions  is  retrieved  through 
the  use  of  one  or  more  of  the  indexing  schemes  (see  subsection  IV.  D). 
The  use  of  an  index  will  normally  retrieve  only  specific  sections  of  a 
system  description.  Because  complete  understanding  of  a  specific 
section  may  entail  examining  other  sections,  it  was  preferable  to 

organize  system  descriptions  by  system,  with  all  information  on  a 

system  grouped  together. 

The  information  contained  in  a  system  description  can  be  broken 
into  the  two  broad  categories  of  descriptive  or  explanative  information. 
Descriptive  information  defines  the  magnitude  and  nature  of  the  subject 
system,  while  explanative  information  gives  reason  for  occurrence  of 
problems  and  attributes  of  the  system.  Normally,  in  retrieving  infor¬ 
mation  from  a  system  description  for  purposes  of  evaluation,  one  is 
primarily  interested  in  the  explanative  information  as  clarified  and  put 
in  context  by  the  descriptive  information.  Purely  descriptive  data  per¬ 
taining  to  cost  factors  are  obtained  through  the  use  of  the  cost  estimating 
iso-graphs. 

D.  Indexes  and  Use 


Indexes  were  developed  for  use  in  retrieving  information  from  the 
system  descriptions.  The  twelve  indexes  fall  into  two  categories:  (1) 
continuous  workload  descriptor  indexes  and  (2)  discrete  system  attri¬ 
bute  indexes. 

1 .  Continuous  Workload  Descriptor  Indexes 

The  Development  Experience  Index  and  the  Operations  Ex¬ 
perience  Index  are  both  based  on  the  use  of  workload  descriptors  for 
retrieval  of  relevant  experience  data.  The  Development  Experience 
Index  uses  workload  descriptors  that  are  causally  related  to  develop¬ 
ment  cost  and  problems.  The  following  workload  descriptors  were 
found  to  be  most  suitable  for  retrieval  of  development  experience: 
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o  Number  of  input  transaction  types 
o  Number  of  input  data  fields 

o  Number  of  output  formats 

o  Number  of  data  base  record  types 

The  workload  descriptors  provided  measures  of  the  size  of  devel¬ 
opment  effort.  Therefore,  development  experience  data  retrieved  by 
this  index  would  relate  to  problems  caused  by  the  size  of  the  develop¬ 
ment  effort.  An  example  of  the  type  of  experience  data  retrieved  by 
this  index  could  be  the  necessity  of  establishing  formal  lines  of  com¬ 
munication  between  analysts  and  programmers  for  systems  of  the  size 
being  evaluated. 

The  Operations  Experience  Index  scheme  was  quite  similar  in 
structure  to  the  Development  Experience  Index.  The  workload  descrip¬ 
tors  that  are  causally  related  to  operations  cost  and  problems  were  the 
following: 

o  Characters  per  month  of  input  volume 

o  Characters  per  month  of  output  volume 

o  Characters  in  data  base 

These  workload  descriptors  provided  a  measure  of  the  size  of  operations 
effort.  Therefore,  operations  experience  data  retrieved  by  this  index 
would  relate  to  problems  caused  by  the  size  of  the  operations. 

The  construction  of  the  Development  Experience  Index  is  described 
in  the  following  paragraphs.  A  similar  construction  was  used  for  the 
Operations  Experience  Index.  The  complete  range  of  each  of  the  four 
workload  descriptors  used  in  the  Development  Experience  Index  was 
represented  by  separate  parallel  logarithmic  scales.  See  Figure  14 
for  an  example  of  these  scales.  On  each  scale,  the  sampled  value  of 
the  workload  descriptor  for  each  of  the  18  systems  was  marked  and 
labeled.  Logarithmic  scales  were  chosen  for  the  same  reasons  as  for 
the  transformation  of  the  original  variable  values  (see  subsection  III.  B.  5). 

To  find  relevant  systems,  a  transparent  index  card  with  slides  for 
operations  indexing  and  development  indexing  was  constructed.  The 
slides  are  marked  with  tolerance  bands,  designated  by  ranking  numbers, 
that  represent  a  fixed  percentage  difference  from  the  proposed  value  of 
a  workload  descriptor.  The  tolerance  bands  and  their  weights  are  as 
follows: 
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Rank 


Tolerance  From 
Proposed  Value 

±  1  5  percent  3 

±30  percent  2 

±45  percent  1 

The  tolerance  bands  were  determined  empirically  by  examining  systems 
selected  with  different  tolerance  bands.  The  different  width  of  toler¬ 
ance  bands  for  the  operations  slide  and  development  slide  results 
from  the  different  logarithmic  scales  used  with  the  slides. 

The  development  slide  is  used  by  centering  it  over  the  proposed 
value  of  Number  of  Input  Transaction  Types  and  entering  in  the  ranking 
table  the  rank  of  all  systems  bounded  by  the  tolerance  bands  on  the 
slide.  This  is  repeated  for  the  other  workload  descriptors.  The  total 
rank  for  a  system  is  determined  by  adding  that  system's  rank  for  each 
individual  workload  descriptor.  The  relevancy  of  systems  is  in  order 
of  total  rank,  with  the  system  of  highest  total  rank  having  greatest 
relevancy. 

2.  Discrete  Attribute  Indexes 


Ten  different  indexes  were  available  for  retrieving  relevant 
data  based  on  discrete  system  attributes: 

o  Functional  Area  Index 

o  Decentralized  Operations  Index 

o  Multiple  Application  Index 

o  Programming  Language  Index 

o  Processing  Type  Index 

o  File  Conversion  Index 

o  Direct  Access  Storage  Index 

o  Computer  Cost  Index 

o  Computer  Index 

o  Security  Index 

With  these  indexes,  all  18  systems  were  classified  into  three  or  more 
categories  based  on  the  attribute  defined  by  the  index  name.  The  attri¬ 
bute  of  the  proposed  system  was  used  to  isolate  all  systems  in  the  same 
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FIGURE  14  -  WORKSHEET  FOR  DEVELOPMENT  EXPERIENCE  INDEX 


category  through  the  use  of  an  index  table.  The  isolated  systems  will 
contain  relevant  experience  data  in  the  area  of  the  category  retrieved. 
Thus,  if  the  proposed  system  used  COBOL,  all  sampled  systems  in  the 
COBOL  category  would  be  retrieved  from  the  programming  language 
attribute  index. 

E.  ADP  Experience  Handbook  Integration  and  Use 

Following  the  development  of  cost  estimation  iso-graphs,  system 
descriptions,  and  indexing  methods,  the  handbook  sections  were  organ¬ 
ized  and  integrated  into  an  easily  usable  form.  Instructions  for  the  use 
of  the  iso-graphs  in  cost  prediction  are  given  in  one  section,  and  an 
index  is  provided  for  retrieving  relevant  information  from  the  system 
descriptions.  Terms  used  in  the  handbook  are  defined  in  a  glossary. 

The  integration  of  these  sections  resulted  in  a  self-sufficient 
handbook.  A  Primer  for  the  use  of  the  handbook  was  developed  to  pro¬ 
vide  an  elementary  text  that  can  be  used  to  train  potential  users  of  the 
handbook.  The  Primer  contains  instructions  for  submission  of  ADPS 
proposals,  a  sample  ADPS  proposal,  and  an  evaluation  of  the  sample 
ADPS  proposal. 

The  Air  Force  ADP  Experience  Handbook  (Pilot  Version)  and  the 
Primer  for  the  Air  Force  ADP  Experience  Handbook  comprise  two 
separate  volumes;  refer  to  PRC  documents  R-930  and  R-931, 
r  e  spectively. 

F.  Findings  and  Conclusions 


1 .  Findings 

o  Iso-graphs  are  an  effective  aid  to  the  manual  solution 
of  cost  estimation  equations. 

o  Macrodescriptions  of  ADP  systems  can  be  written. 

o  The  indexing  attributes  are  as  follows: 

Workload  descriptors  (small  sensitive  set 
of  seven) 

Functional  area 

Decentralized  operations 

Multiple  applications 

Programming  language 

Processing  type 
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File  conversion  type 

Direct  access  storage  capacity 

Computer  cost 

Computer  make  and  model 

Security  requirements 

o  Iso-graphs,  macrodescriptions,  and  indexes  are 
organizable  into  a  usable  handbook. 

2.  Conclusions 


The  ADP  Experience  Handbook  will  be  a  useful  tool  for  per¬ 
sonnel  concerned  with  management  of  ADP  with  the  Air  Force. 
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V.  PHASE  III  PLANNING 


The  purpose  of  this  section  is  to  briefly  review  the  planning  ac¬ 
tivity  that  preceded  the  actual  development  of  the  operational  concept 
and  development  plan  for  Phase  III.  The  activities  of  data  collection, 
data  analysis,  and  experience  handbook  development  to  the  midpoint 
of  Phase  II  have  led  to  preliminary  conclusions.  When  integrated, 
these  conclusions  indicated  that  Phase  III  will  be  desirable  to  the  Air 
Force  if  it  is  feasible  and  cost  effective.  Hence,  the  Phase  III  oper¬ 
ational  concept,  development  plan,  and  the  analysis  of  costs  and  bene¬ 
fits  were  developed  and  are  covered  in  detail  in  Volume  III  of  this  final 
report. 

The  primary  goal  of  the  Phase  III  planning  activity  was  to  estab¬ 
lish  a  concept  for  collecting,  editing,  reducing,  and  using  data  process¬ 
ing  data  at  HQ  USAF.  It  was  therefore  necessary  to  learn  more  about 
who  used  such  data  and  for  what  purpose,  who  reported  such  data  and 
in  what  detail,  what  was  the  general  content  of  an  ADPS  proposal,  and 
what  evaluation  processes  took  place,  etc. 

Accordingly,  project  staff  members  visited  the  Air  Staff  on  sev¬ 
eral  occasions  to  collect  such  information,  principally  to  determine  in 
detail  Data  Automation  Proposal  (DAP)  evaluation  procedures  and  pol¬ 
icies  as  well  as  ADP  experience  reporting  procedures.  After  lengthy 
discussions  with  members  of  AFADA  and  AFSPD  and  after  analysis  of 
all  appropriate  regulations,  manuals,  and  operating  instructions,  a 
general  picture  of  ADPS  proposal  procedures  and  reporting  procedures 
was  established.  These  findings  are  summarized  in  Appendixes  F  and  G. 

It  became  evident  during  this  task  that  Data  Automation  Proposals 
are  only  one  of  several  types  of  ADPS  proposals  that  must  be  judged  by 
HQ  USAF  and,  further,  that  there  are  several  places  in  HQ  USAF  that 
the  judging  takes  place.  In  addition  to  DAP's,  the  following  documents 
can  propose  systems:  Required  Operational  Capability  (ROC);  Require¬ 
ments  Action  Directive  (RAD);  Advance  Communications  --  Electronic 
Requirements  Plan  (ACERP);  Communications  --  Electronics  Imple¬ 
mentation  Plan  (CEIP);  Program  Change  Proposal  (PCP);  Proposed 
System  Package  Plan  (PSPP);  System  Package  Plan  (SPP);  Preliminary 
Technical  Development  Plan  (PTDP);  and  others.  Essentially,  all  parts 
of  the  Air  Staff  can  become  involved  in  the  evaluation  of  a  proposal; 
however,  the  designated  offices  of  primary  responsibility  (OPR's)  in¬ 
clude  the  Director  of  Data  Automation  (AFADA),  the  Director  of  Pro¬ 
duction  (AFSPD),  the  Director  of  Maintenance  Engineering  (AFSME), 
the  Director  of  Operational  Requirements  and  Development  Plans 
(AFRDQ),  and  the  Assistant  for  Research  and  Development  Pro¬ 
gramming  (AFRRP). 
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The  scope  of  the  Phase  III  planning  activity  was  broadened  to 
ensure  that  the  concept  established  for  implementation  in  Phase  III  of 
the  project  met  the  ADP  management  needs  of  all  parts  of  the  Air  Staff. 
Other  Air  Staff  ADP  management  functions  include  efficient  utilization 
of  ADP  assets,  prosecution  of  ADP  standards  programs,  control  of 
on-going  ADP  developments  and  operational  systems,  forecasting  of 
ADP  budgets,  and  performance  of  special  studies  of  Air  Force  ADP. 

The  preliminary  conclusion  of  Phase  III  planning  is  that  the  cen¬ 
tral  feature  of  Phase  III  should  be  an  Air  Force  ADP  management  in¬ 
formation  system  capable  of  systematically  collecting,  editing,  storing, 
retrieving,  and  putting  to  use  experience  and  asset  data  from  all  Air 
Force  ADP  systems  and  data  processing  installations.  Although  the 
principal  need  for  the  system  is  at  HQ  USAF,  other  Air  Force  organi¬ 
zations  could  make  use  of  it  (e.  g.  ,  SPO's). 


60 


APPENDIX  A 

AIR  FORCE  AND  CIVILIAN  PERSONNEL 
CONTACTED  DURING  DATA  COLLECTION 
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Appendix  A  provides  a  list  of  the  personnel  from  whom  data 
were  collected  at  the  18  installations  interviewed.  Names  of  PRC 
interviewers,  location  of  the  installation,  and  dates  of  interviews 
are  also  provided. 
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APPENDIX  B 

DATA  COLLECTION  QUESTIONNAIRE 


73 


Appendix  B  consists  of  the  data  collection  questionnaire  as  re¬ 
vised  after  the  pilot  data  collection  effort  on  the  Military  Personnel 
Center  Personnel  Data  System--Officer  (MPC  PDSO)  at  Randolph  Air 
Force  Base. 

Parts  A,  B,  C,  D,  and  E  comprise  the  questionnaire  proper, 
and  Part  F  (Summary  Sheet  for  Numerical  Data)  is  a  summary  of  re¬ 
duced  data  for  the  statistical  analysis. 
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QUESTIONNAIRE  OUTLINE 


A.  General  Description 

B.  Proposal 

1.  Content 

2.  Preparation 

C.  Development 

1.  Technical  Approach 

2.  Management  Approach 

3.  Schedule 

4.  Manpower 

5.  Hardware 

6.  Software  Supplied  by  Others 

7.  System  Design 

8.  Programming 

9.  File  Conversion 

D.  Operations 

1.  Organization 

2.  Manpower 

3.  Workload 

4.  Scheduling 

5.  Utilization 

6.  Program  Maintenance 

7.  Hardware 

8.  Support  Programming 

9.  Facility 

E.  Future  Plans 

F.  Summary  Sheet  for  Numerical  Data 
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A.  GENERAL  DESCRIPTION 


1. 


2. 

3. 

4. 


5. 


6. 


7. 


8. 


9. 


1. 


Obtain  an  organization  chart  showing  position  of  both  ADPS  and 
prime  user(s)  within  the  Air  Force. 

Describe  the  mission  of  the  ADPS. 

Describe  the  mission  of  the  prime  user(s). 

Obtain  the  DSAP  symbols  and  titles  (see  RCS  8-AF-E6)  for  the 
ADPS. 

Obtain  a  simplified  functional  block  diagram  and  succinctly  de¬ 
scribe  the  work  the  ADPS  does. 

Develop  a  narrative  history  of  ADPS  from  inception  to  present  day. 
What  are  the  consequences  of  ADPS  being  down  for  an  extended  pe¬ 
riod  of  time?  (Include  backup  capability.) 

Describe  any  security  factors  involving  ADPS  and  installation. 
Describe  any  problems,  consequences  of  problems,  and  expected 
solutions. 

B.  PROPOSAL 


Content 


Describe  the  content  of  the  DAP  (or  other  paperwork  or  briefing) 
upon  which  Hq.  USAF  based  approval  of  ADPS.  Include  the  pro¬ 
poser's  conception  of  future  events  and  activities  during  develop¬ 
ment  and  operations,  using  the  following  topics  as  guidelines: 

_ Development _  _ Ope  rations _ 


Tasks  to  be  performed  during 
development 

Organizational  approach 

Schedule 

Manpower 

Hardware 

Software  supplied  by  others 


Organization 

Manpower 

Workload 

Files 

Scheduling 

Utilization 

Program  maintenance 
Hardware 

Systems  programming 
Facility 


78 


2. 


Preparation 


Describe  preparation  effort  for  proposal;  for  example,  who  did  it, 
how  effort  was  organized,  and  pertinent  dates. 

C.  DEVELOPMENT 

1 .  Technical  Approach 

Describe  steps  or  activities  undertaken  during  development,  par¬ 
ticularly  in  contrast  to  what  was  proposed. 

2.  Management  Approach 

a.  Describe  or ganizational  approach  to  development;  for  example, 
tasks  assigned  to  various  organizational  entities  and  relation¬ 
ships  among  analysts -programmer s -user s -management. 

b.  Describe  management  control  methods  used  during  develop¬ 
ment;  for  example,  PERT,  cost  control,  or  progres s  reports. 

3.  Schedule 

(Use  sheet  provided.) 

4.  Manpower 

a.  Collect  data  on  personnel  buildup  during  development  phase. 
(Use  sheet  provided.) 

b.  See  that  personnel  data  sheets  are  distributed. 

5.  Hardware 

a.  Describe  hardware  selection  process;  for  example,  the  RFQ, 
number  of  bidders,  and  selection  criteria. 

b.  Describe  any  unusual  installation  problems. 

6.  Software  Supplied  by  Others 
(Use  sheet  provided.) 

7.  System  Design 

a.  Was  this  a  pioneering  application?  If  so,  why? 

b.  What  is  the  system  design?  Obtain  a  flow  chart. 
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c.  What  workload  was  the  system  designed  to  handle?  Is  it 
different  from  the  workload  at  the  proposal  stage  and  the 
present  workload? 

d.  To  what  extent  did  the  system  design  change  during  the  de¬ 
velopment  stage? 

e.  Were  there  interface  problems  with  other  ADP  systems? 

f.  Did  earlier  automation  efforts  make  this  development 
easier  ? 

g.  Describe  documentation  activities  with  regard  to  system 
specification  to  both  programmers  and  users,  and  design 
changes. 

8.  Programming 

a.  In  your  opinion,  was  the  machine  "mature11  at  the  time  of 
system  development? 

b.  Describe  documentation  activities  of  programmers  with  re 
gard  to  program  specifications,  detailed  flow  charts,  pro¬ 
gram  changes,  and  manuals  for  operators,  users,  and  pro 
gram  maintenance  personnel. 

c.  Acquire  a  list  of  all  programs  in  the  system  and  classify 
them  according  to  the  following  primary  program  functions 

(1)  Input  Edit  (input  conversion,  data  edit,  error  and 
logic  checks) 

(2)  File  Maintenance  (file  update,  extract  data) 

(3)  Report  Generation  (data  edit,  print,  display) 

(4)  Merge  (sequence  ordered  sets  of  data) 

(5)  Compute  (arithmetic) 

(6)  Sort  (sequence  unordered  sets  of  data) 

(7)  Query-File  Search  (search  file  for  desired  items, 
display  items) 

(8)  Control  (job  scheduling,  priority  handling,  hardware 
component  assignments) 

(9)  Support  (nonapplication  programs) 

(Use  sheets  provided.) 
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d.  Were  any  unusual  programming  techniques  employed  (for 
example,  list  processing  or  dynamic  memory  allocation)? 

e.  Describe  compilation  and  checkout  activities.  Include  lo¬ 
cation  and  type  of  machine,  use  of  emulator / simulator, 
whether  shop  was  open  or  closed,  usual  turnaround  time, 
and  whether  special  input  data  were  developed  for  checkout 
purposes. 

f.  Describe  system  test  activities. 

g.  How  many  computer  hours  were  required  for  development 
of  this  ADPS?  Include  compilation,  assembly,  checkout, 
and  tests. 

9.  File  Conversion 

Describe  file  conversion  activities.  Include  storage  media  before 
and  after,  how  "clean"  the  files  were,  manual  transcription  and 
editing  requirements,  manpower  used,  special  programming  re¬ 
quirements,  and  computer  time  used. 

D.  OPERATIONS 

1 .  Organization 

Obtain  an  organization  chart  for  the  personnel  operating  the  ADPS. 

2.  Manpower 

a.  How  many  programmers  are  in  program  maintenance? 

b.  How  many  people  are  in  computer  operations?  (Include  com¬ 
puter  operators,  schedulers,  production  control,  tape 
librarians. ) 

c.  How  many  people  are  in  EAM  operations?  (Include  input 

batching,  keypunch,  tab  operators.) 

d.  See  that  Operations  Personnel  Data  Sheets  are  distributed. 

3.  Workload 

Complete  workload  sheets. 
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4. 


Scheduling 


a.  Is  there  a  master  schedule?  If  yes,  block  in  master  sched¬ 
ule  diagram.  (Use  sheet  provided.)  Include  maintenance 
and  program  testing. 

b.  Is  a  daily  schedule  prepared?  If  yes,  describe  procedure 
and  when. 

c.  Describe  procedures  for  scheduling  monthly,  quarterly, 
yearly,  and  special  reports  or  runs. 

d.  Describe  effect  and  impact  of  system  monitor  or  control 
program  on  scheduling  function. 

e.  Describe  procedure  for  handling  priorities. 

f.  Describe  any  other  functions  of  scheduling. 

g.  Is  shop  open  or  closed? 

5.  Utilization 

a.  Obtain  enough  information  to  construct  a  "pie  chart”  for  a 
typical  day's  utilization  of  the  machine.  Include  time  for 
program  maintenance,  downtime,  production,  and  compila¬ 
tion,  or  however  the  installation  breaks  the  time  down.  The 
RCS  6-AF-E6  and  8-AF-E6  reports  will  be  useful  for  this 
purpose.  If  these  reports  do  not  exist,  other  reports  titled 
"program  run  analysis”  or  "computer  usage  report"  may  be 
obtainable. 

b.  Obtain  the  following  for  the  application  under  study: 

Machine 

Hours  Per  Month 

Input  edit 

File  maintenance 

Report  generation 

Me  r  ge 

Sort 

Compute 

Query 

Program  maintenance 


82 


c.  The  people  working  in  operations  and  the  nonproductive 

machine  time  must  be  apportioned  to  the  application  under 
study.  If  the  application  under  study  shares  the  machine 
with  other  applications,  obtain  enough  information  to  make 
the  apportionment  on  the  basis  of  machine  hours  utilized 
for  the  application. 

6.  Program  Maintenance 

a.  What  percent  of  the  people  involved  in  program  maintenance 
were  in  the  original  system  development? 

b.  What  percent  of  program  maintenance  can  be  considered  to 
be  corrections  and  what  percent  improvements? 

c.  What  type  of  documentation  is  produced  for  program  changes? 

d.  What  is  the  average  turnaround  time  for  checkout  work? 

7.  Hardware 

a.  Obtain  enough  information  to  diagram  the  current  hardware 
configuration  including  manufacturers  and  model  numbers 
for  each  component.  Use  the  RCS  6-AF-E6  report,  if 
available. 

b.  How  has  hardware  configuration  changed  since  the  system 
was  declared  operational? 

c.  What  is  monthly  rental  or  original  purchase  price? 

d.  Comment  on  the  reliability  of  the  hardware. 

8.  Support 

a.  Describe  current  systems  programming  activities.  (Sys¬ 
tems  programming  is  the  maintenance  and  development  of 
compilers,  assemblers,  control  programs,  and  utility 
routines. ) 

b.  Describe  data  storage  activities  including  tape  libraries, 
card  libraries,  and  physical  handling  methods. 

c.  Describe  PCAM  and  keypunch  activities. 
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9. 


Facility 


a. 

b. 

c. 


1.  What 


Is  there  adequate  space  for  movement  of  men  and  materials 
in  the  computer  room? 

Do  operators  have  good  visibility  of  equipment  status  indica¬ 
tors  and  peripheral  equipment? 

Does  standby  power  exist? 

E.  FUTURE  PLANS 

is  planned  for  the  future  ? 
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DEVELOPMENT  PERSONNEL  BUILDUP  BY  CATEGORY 


DATA  SHEET  FOR  DEVELOPMENT  PHASE  MANAGERIAL  AND  LEAD  PERSONNEL 
(Please  fill  out  your  line  and  pass  the  sheet  on) 


Name 

Rank 

or 

GS  No. 

Job  Title 

Education 

Experience  (Yra) 

High  school 

graduate  ? 

Years 

college 

Degree(s) 

In  data 

processing 

In  using  this 

language 

In  the  field  of: 
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DATA  SHEET  FOR  DEVELOPMENT  PHASE  PERSONNEL 


(Please  fill  out  your  line  and  pass  the  sheet  on) 


Name 

Rank  or 
GS  No. 

Job  Title 

No.  of  Years  Experience 

In  Data 
Processing 

In  Field  of: 
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SOFTWARE  SUPPLIED  BY  OTHERS 
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CHARACTERISTICS  OF  PROGRAMS 


Number  of 
Machine 
Instructions 

Number  of 
Source 
Statements 

language 
in  Which 
Program 
Was  Written 

Developed  by 
Manufacturer, 
Contractor, 
or  AF 

Program 
Function 
(Sort,  Merge 
Input  Edit,  etc.) 

Program  Name 

Program 

Designation 
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DATA  SHEET  FOR  OPERATIONS  PHASE  MANAGERIAL  AND  LEAD  PERSONNEL 
(Please  fill  out  your  line  and  pass  the  sheet  on) 


Name 

Rank 

or 

GS  No. 

Tob  Title 

Education 

Experience  (Yrs.) 

High  school 

graduate  ? 

Years 

colie  ge 

De  gree(s) 

In  data 

process ing 

In  using  this 

lan  guage 

In  the  field  of* 
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DATA  SHEET  FOR  OPERATIONS  PHASE  PERSONNEL 


(Please  fill  out  your  line  and  pass  the  sheet  on) 


Nam  e 

Rank  or 
GS  No. 

Job  Title 

No.  of  Years  Experience 

In  Data 
Proces  sing 

In  Field  of: 
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INPUTS 


Ave.  % 
of  rec¬ 
ords  re- 
j  ected 

by 

input 

edit 

Max. 
freq.  of 
a  r  rival 
(At  com¬ 
puter 
(Specify 
units) 

Ave  rage 
freq.  of 
a  r rival 
(At  com¬ 
puter) 
(Specify 
units) 

No.  of 
types  of 
trans  - 

actions 

No.  of 
unique 
fields 

Max.  no. 
of  char, 
per 

record 

Max.  no. 
of 

records 
per  mo. 

Ave.  no. 
of  char, 
per 

record 

Ave.  no. 
of 

records 
per  mo. 

Batched 

o  r 

direct? 

(At 

computer) 

Source 

Description 
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FILES 


Include 

in  nonre  - 

dundant 

set? 

Fre¬ 

quency 

of 

use 

Fre¬ 

quency 

of 

update 

^  of 

fields 

which 

have 

variable 

length 

entries 

Net  growth 
per  month 
(specify 
units) 

If  applicable 

Max.  no. 
of  char, 
per 

record 

Max.  na 
of 

records 

Ave.  no. 
of  char, 
per 

record 

Ave.  no. 

1  of 

records 

Storage 

medium 

Description 
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Note:  "Variety"  is  the  number  of  files  in  nonredundant  set. 


OUTPUTS 


1 

Response 

'  time 

No.  of  diff, 
types  of 
units 
(If  not  ail 
same 
format) 

Max.  no.  of 
char, 
per  unit 

Max.  no.  of 
units 

per  month 

! 

Ave.  no.  of 
char, 
per  unit 

i 

Ave.  no.  of 
i  units 

1  pe  r  month 

Batched 
or  direct0 
(At 

compute  r) 

Recipient 

Desc  ription 
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MASTER  SCHEDULE 


— - 1 - — 

Mon  Tues  Wed  Thurs  Fri  .  Sat  Sun 

1 

0 

1 

1 

i 

i 

1 

2 

1 

1 

| 

3 

i 

4 

1 

1 

5 

i 

i 

6 

1 

i 
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i 

8 

i 

9 

i 

i 

10 

i 

1  1 

i 

12 

i 

i 

13 

1 

14 

1 

1  5 

i 

■ - * - 

1  6 

1 

17 

1 

ft 

18 

i 

i 

19 

i 

20 

i 

i 

21 

i 

22 

i 

l 

23 

i 

i 
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F.  SUMMARY  SHEET  FOR  NUMERICAL  DATA 


System _  Location _ Analyst 

Date 


Name  of  Variable 

Must 

Collect 

Collect  if 
Applicable 

Collect 
if  Easily 
Obtainable 

V alue  of 
Variable  or 
Remarks 

Independent  variables  for 

regression  analysis 

Input  -  Batched 

1.  Ave.  volume p er  month 

X 

2.  Max.  volume  per  month 

X 

^  Ave.  frequency  per  hour 

X 

4.  Max.  frequency  per  hour 

X 

5.  Variety]  (No.  types  of 

transactions) 

X 

6.  Variety-,  (No.  unique  data 

£ 

fields) 

X 

7.  Variety^  (No.  unique 

J 

parameter  fields) 

X 

8.  Reliability 

X 

Input  -  Unbatched 

X 

9.  Ave.  volume  per  month 

10.  Max.  volume  per  month 

X 

11.  Ave.  frequency  per  hour 

X 

12.  Max.  frequency  per  hour 

X 

13.  Variety  j  (No.  types  of  

transactions) 

X 

14.  Variety^  (No.  unique  data  

fiel  H«) 

X 

15.  Variety^  (No.  unique 

parameter  fields) 

X 

16.  Reliability 

X 
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F.  SUMMARY  SHEET  FOR  NUMERICAL  DATA 
(Continued) 


System _  Location _ Analyst 

Date 


Name  of  Variable 

Must 
Col  lect 

Collect  if 
Applicable 

Collect 
if  Easily 
Obtainable 

Value  of 
Variable  or 
Remarks 

Output  -  Indirect 

17. 

Ave.  volume  per  month 

X 

18. 

Max.  volume  per  month 

X 

19. 

Variety  (No.  report  formats) 

X 

Res  do  n  set  i  me 

X 

QuLnuL  -  Direct  _ 

21. 

Ave.  volume  per  month 

X 

22. 

Max.  volume  per  month 

X 

23. 

Variety  (No.  report  formats) 

X 

24. 

Response  time 

X 

Data 

Base 

25. 

Average  size 

X 

26. 

Maximum  size 

X 

27. 

Net  growth  per  month 

X 

28. 

%  of  data  updated  per  month 

X 

29. 

Variety  (No.  types  of 

records) 

X 

30. 

No.  items  "kept  track  of" 

X 

3L 

Net  growth  per  month  of 

 ...  .  items  "kept. track  of" 

X 

Processing  Functions  -%of 

instructions  fnr; 

32. 

Input  edit 

X 

33. 

File,  ma  intena  nc  e 

X 

34. 

Report  generation 

X 

35. 

Merge _ 

_ X _ 
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F.  SUMMARY  SHEET  FOR  NUMERICAL  DATA 
(Continued) 


System _  Location _  Analyst 

Date 


Name  of  Variable 

Must 

Collect 

Collect  if 
Applicable 

Coll  cc  t 
if  Easily 
Obtainable 

Value  of 

V ar iable  or 
Remarks 

Processing  Functions  (Cont'd) 

36.  Sort 

X 

37.  Compute 

X 

38.  Query 

X 

Personnel  -  Development 

Manage  rs 

39.  Ave.  ranks/GS 

X 

40.  Ave.  years  college 

X 

41.  Ave.  years  in  ADP 

X 

42.  Ave.  vears  in  functional  area 

X 

Analysts 

43.  Ave.  rank/GS 

X 

44.  Ave.  years  college 

X 

45.  Ave.  years  in  ADP 

X 

46.  Ave.  years  in  functional  area 

X 

Programmers 

47.  Ave.  rank/GS 

x 

.48.  Ave.  years  college 

X 

4Q .  Ave.  vears  in  ADP 

X 

50.  Ave.  vears  with  language 

X 

51.  Ave,  vears  in  functional  area 

X 

Personnel  -  Operations 

52.  Ave.   rank/GS ...  

X 

53.  Ave.  years  college 

.  -  X- 

54.  Ave.  years  in  ADP 

X 

-55.  ..Ay£ _ y ear s-iniunctional area 

X 
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F.  SUMMARY  SHEET  FOR  NUMERICAL  DATA 
(Continued) 


System _ Location _ _ Analyst^ 

Date 


Name  of  Variable 

Must 

Collect 

Collect  if 
Applicable 

Collect 
if  Easily 
Obtainable 

Value  of 
Variable  or 
Remarks 

Hardware 

56.  Date  of  first  delivery  (from 

Adams  Associates)* 

X 

57.  Planning  information  (any 

input,  output,  data  base 

—  or  processing function 

variables  stated  at  time 

of  proposal) 

X 

Dependent  variables  for 

regression  analysis 

P ersonnet  cost 

101.  Man-months  fox  design  and 

implementation  

X 

102,  No,  people  in  program  de- 

velopment  and 

 maintenance  

X 

103.  No.  people  in  computer 

operations 

X 

104.  Development  time 

X 

Planned  and  actual  dates  for: 

105.  Hardware  installation 

X 

106.  Compiler /assembler 

operating 

V 

107.  Executive  operating 

V 

operating 

X 

Must  be  combined  with  other  variables  in  regression  analysis. 
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F.  SUMMARY  SHEET  FOR  NUMERICAL  DATA 
(Continued) 


System 


Location 


Analyst 

Date 


Name  of  V a r i a bl o 

Must 
Coll  ect 

Collect  if 
Applicable 

Collect 
if  Easily 
Obtainable 

Value  of 
Variable  or 

R  c  m  a  r  k  s 

109. 

Application  operational 

X 

Hardware  cost  -  installation  as 

a 

whole  (hour s/ month) 

110. 

Application  production 

X 

111. 

Application  preparation 

X 

112, 

Procram  development  and 

maintenance 

X 

113. 

Total  chargeable  lost  time 

X 

1  14. 

Total  operational  use 

(L  110  tb r ouch  114) 

X 

115. 

Total  nonchar  reable  lost 

time 

X 

U6. 

Monthly  rental.  $ 

X 

For 

application: 

117. 

Production  hours/month 

X 

%  of  production  allocatable 

118. 

Input  edit 

X 

119. 

File  maintenance 

X 

120. 

Report  peneration 

X 

121. 

Merg£ 

X 

1 22. 

Sort  ...  

X 

123. 

Compute 

X  

1 24. 

..  Query  .  .   .   

X 

125. 

Program  development  and 

maintenance  hrs /month 

X 

l  -  -  - 
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F.  SUMMARY  SHEET  FOR  NUMERICAL  DATA 
(Continued) 


System 


Location 


Analyst 

Date 


Name  of  Variable 

Must 

Collect 

Collect  if 
Applicable 

Collect 
if  Easily 
Obtainable 

Value  of 
Variable  or 
Remarks 

126.  Hours  for  compilation. 

assembly,  checkout,  svs- 

tern  test  during  develop- 

ment  phase 

X 

Processing  functions  ... 

127.  Total  object  instruction 

in  application 

X 

128.  Total  source  statements 

in  application 

X 

129.  Total  object  instructions 

in  executive 

X 

130.  Planning  information  (anv 

personnel  cost,  develop- 

ment  time,  hardware  cost. 

or  processing  function 

variables  stated  at  time  of 

proposal) 

X 

Must  be  combined  with  other  variables  in  regression  analysis. 
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APPENDIX  C 

DEFINITIONS  OF  FACTORS 
AND  DESCRIPTORS 
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Dependent  Variables 


A. 


Symbol  N  a  me 


Definition 


Y 


1 


Y 


2 


Y 


3 


Y 


4 


Y 


5 


Y 


6 


Y 


7 


Man-months  of 
development  effort 


Dollars  of  hardware 
cost  for  program 
checkout 


The  number  of  man-months  expended 
by  all  relevant  personnel  including 
managers,  analysts,  programmers, 
and  operators  to  develop  the  ADPS 
during  the  development  phase  which 
begins  with  the  start  of  system  design 
and  ends  when  the  system  is  declared 
operational.  During  this  develop¬ 
ment  phase,  such  activities  as  de¬ 
tailed  system  design,  programming, 
checkout,  and  equipment  installation 
are  accomplished. 

The  hardware  cost  for  computer  hours 
used  for  program  checkout  during  the 
development  phase  of  the  ADPS. 


Number  of  program 
maintenance  personnel 


Number  of  operations 
personnel 


Dollars  per  month  of 
hardware  cost  for  ap¬ 
plication  production 


Dollars  per  month 
of  hardware  cost  for 
program  maintenance 


Months  of  elapsed 
development  time 


The  number  of  personnel,  including 
managers,  analysts,  and  program¬ 
mers,  involved  in  improving,  chang¬ 
ing,  and  correcting  programs  of  a 
system  during  the  operations  phase. 

The  number  of  related  personnel,  in¬ 
cluding  operator  s  ,  schedulers,  data 
edit  personnel,  magnetic  tape  librar¬ 
ians,  report  binder  s ,  managers,  etc., 
used  to  process  the  ADPS  programs 
on  the  computer  during  the  operations 
phase . 

The  hardware  cost  for  monthly  com¬ 
puter  hours  charged  to  the  user  of 
the  ADPS  for  processing  that  is  not 
of  a  developmental  or  corrective 
nature . 

The  hardware  cost  for  monthly  com¬ 
puter  hours  used  for  processing  im¬ 
provements,  changes,  and  corrections 
to  programs  of  an  operational  ADPS. 

The  number  of  calendar  months  elapsed 
from  the  date  system  design  for  the 
ADPS  is  begun  to  the  date  it  is  de¬ 
clared  operational. 
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Symbol  Name 


Definition 


Y 


8 


Y 


9 


Y 


10 


Y 


11 


Y 


12 


Y 


13 


Y 


14 


Y 


15 


Number  of  source 
statements 


Number  of  object 
instructions 


Percent  of  produc¬ 
tion  hours  for 
input  edit 


Percent  of  produc¬ 
tion  hours  for  file 
maintenance 


Percent  of  produc¬ 
tion  hours  for 
report  generation 


Percent  of  produc¬ 
tion  hours  for  merge 


Percent  of  produc¬ 
tion  hours  for  sort 


Percent  of  produc¬ 
tion  hours  for 
compute 


The  number  of  lines  of  code  written 
by  the  programmer  in  any  source 
language  for  the  ADPS.  This  may  be 
the  same  as  the  number  of  instruc¬ 
tions  in  machine  language. 

The  number  of  instructions  generated 
by  the  compiler  or  assembler  for  the 
ADPS.  This  is  the  number  of  machine- 
format  instructions  in  an  object  pro¬ 
gram  deck  that  can  be  processed 
directly  by  the  computer. 

The  percent  of  production  hours  per 
month  for  input  edit  where  input 
edit  is  performed  on  input  data  to 
prepare  it  for  the  primary  processing; 
e.g.,  limit  and  logic  checking,  field 
conversion,  and  data  edit. 

The  percent  of  production  hours  per 
month  for  file  maintenance  where 
file  maintenance  is  the  modification 
of  a  file  to  incorporate  corrections, 
additions,  and  deletions. 

The  percent  of  production  hours  per 
month  for  report  generation  where 
report  generation  is  the  transforma¬ 
tion  of  results  from  primary  compu¬ 
tations  to  outputs  for  the  system 
user. 

The  percent  of  production  hours  per 
month  for  merge  where  merge  is 
the  combining  of  items  of  records 
from  two  or  more  sequenced  files 
with  the  same  key  into  one  sequenced 
file. 

The  percent  of  production  hours  per 
month  for  sort  where  sort  is  the  ar¬ 
ranging  of  records  of  information 
according  to  rules  operating  upon 
key(s)  contained  in  the  records. 

The  percent  of  production  hours  per 
month  for  compute  where  compute  is 
the  performance  of  logical,  arithmetic, 
and  decisional  operations  on  data. 
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Definition 


Symbol  Name 


Y 


16 


Y 


17 


Percent  of  produc-  The  percent  of  production  hours  per 
tion  hours  for  query  month  for  query  where  query  is  acting 

on  a  demand  input  which  specifies  that 
data  be  accessed  via  file  search  and 
be  displayed  or  output. 


Percent  of  produc-  The  percent  of  production  hours  per 

tion  hours  for  month  for  control  where  control  is 

control  a  computer  processing  function  that 

expedites  all  other  computer  process¬ 
ing  functions;  e.g.,  job  scheduling, 
priority  handling,  segment  overlaying, 
data  management,  and  hardware  as¬ 
signment,  etc. 


B.  Independent  Variables 


Symbol  Name 


Definition 


X 


1 


X 


z 


X 


3 


X 


4 


X 


5 


X6 


Characters  per 
month  of  input 
volume 


Number  of  input 
transaction  types 


Number  of  input 
data  fields 


Percent  of  input 
r  ejects 


Characters  per 
month  of  output 
volume 


Number  of  output 
formats 


The  expected  amount  of  ADPS  input 
originating  outside  the  ADPS,  meas¬ 
ured  in  characters  per  month.  Inter¬ 
mediate  inputs  of  the  ADPS  should 
not  be  included.  On  unit  record  input, 
only  character  positions  used  for  data 
are  counted. 

A  count  of  different  transaction  types 
of  ADPS  input  which  normally  are 
identified  by  a  unique  transaction 
code  and/or  a  unique  input  format. 

A  count  of  data  fields  from  the  ADPS 
input  that  are  unique  in  content  and/ 
or  format;  e.  g.  ,  if  there  is  a  data 
field  for  name  on  six  different  card 
formats,  the  number  of  unique  data 
fields  is  one. 

Input  data  error  rate  measured  by  the 
ratio  of  the  number  of  rejected  records 
to  the  number  of  expected  records  per 
month  multiplied  by  100. 

The  expected  amount  of  ADPS  output 
destined  to  users,  measured  in  char¬ 
acters  per  month.  Intermediate  out¬ 
puts  of  the  ADPS  are  not  included. 

Only  nonblank  characters  are  counted. 

The  number  of  different  types  and  for¬ 
mats  of  ADPS  outputs. 
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Symbol 

Name 

Definition 

X7 

Characters  in 
data  base 

The  expected  number  of  characters 
in  the  data  base  where  the  data  base 
is  a  collection  of  files  that  contain 
unique  information,  are  accessible 
to  the  ADPS,  and  are  normally  ref¬ 
erenced  or  updated  with  relatively 
high  frequency.  Intermediate  files 
are  not  counted. 

X8 

Number  of  data 
base  record  types 

The  number  of  logical  record  types 
in  the  data  base  where  a  logical 
record  is  a  set  of  logically  related 
data  fields  independent  of  the  physical 
manner  of  storage. 

X9 

Percent  of  source 
statements  for 
input  edit 

The  percent  of  source  statements  for 
input  edit.  (See  for  definition  of 

source  statements  and  Y^q  for  defini¬ 
tion  of  input  edit.  ) 

X  1 0 

Percent  of  source 
statements  for  file 
maintenance 

The  percent  of  source  statements  for 
file  maintenance.  (See  Y^  for  defini¬ 
tion  of  file  maintenance.) 

X11 

Percent  of  source 
statements  for 
report  generation 

The  percent  of  source  statements  for 
report  generation.  (See  Y^ for  defini 
tion  of  report  generation.) 

X12 

Percent  of  source 
statements  for 
merge 

The  percent  of  source  statements  for 
merge.  (See  Y^  f°r  definition  of 
merge.  ) 

X 1 3 

Percent  of  source 
statements  for 
sort 

The  percent  of  source  statements  for 
sort.  (See  Y^  for  definition  of  sort.  ) 

X14 

Percent  of  source 
statements  for 
compute 

The  percent  of  source  statements  for 
compute.  (See  Y^-  for  definition  of 
compute.  ) 

X15 

Percent  of  source 
statements  for 
query 

The  percent  of  source  statements  for 
query.  (See  Y^  for  definition  of 
query. ) 

X 1 6 

Percent  of  source 
statements  for 
control 

The  percent  of  source  statements  for 
control.  (See  Y^  for  definition  of 
control.  ) 

X17 

Average  number  of 
years  of  college 
education  for  devel¬ 
opment  managers 

Development  managers  college  educa¬ 
tion,  measured  in  average  number  of 
years,  where  development  managers 
are  the  individuals  responsible  for 
directing  and  coordinating  all  or  part 
of  the  activities  associated  with  an 
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Symbol  Name 


Definition 


X 


18 


X 


19 


X 


zo 


X 


21 


X 


22 


X 


23 


X 


24 


X 


25 


Average  number  of 
years  of  ADP  expe¬ 
rience  for  develop¬ 
ment  managers 


Average  number  of 
years  of  ADP  expe¬ 
rience  for  analysts 


Average  number  of 
years  of  ADP  expe¬ 
rience  for 
programmers 


Average  number  of 
years  of  ADP  expe¬ 
rience  for  operations 
personnel 

Average  number  of 
years  of  functional 
area  experience  for 
development 
managers 


Average  number  of 
years  of  functional 
area  experience  for 
analysts 

Average  number  of 
years  of  functional 
area  experience  for 
programmers 

Average  number  of 
years  of  functional 
area  experience  for 
operations  personnel 


ADPS  during  the  development  phase. 
Only  managers  devoting  at  least  10 
percent  of  their  time  to  the  system 
are  considered. 

Average  number  of  years  in  the  field 
of  automatic  data  processing  (ADP) 
for  development  managers.  (See  X^ 
for  definition  of  development 
managers .  ) 

Average  number  of  years  in  ADP  for 
analysts,  who  are  persons  skilled  in 
the  definition  of  and  the  development 
of  techniques  for  solving  a  problem. 

Average  number  of  years  in  ADP  for 
programmers,  who  are  persons  who 
prepare  problem  solving  procedures 
and  logical  flow  charts,  and  code  and 
debug  programs. 

Average  number  of  years  in  ADP  for 
operations  personnel.  (See  for 
definition  of  operations  personnel.  ) 


Average  number  of  years  of  experi¬ 
ence  in  a  field  of  application,  such 
as  accounting,  inventory  control, 
weather  forecasting,  etc.  ,  for  devel¬ 
opment  managers.  (See  X ^  for  def¬ 
inition  of  development  managers.  ) 

Average  number  of  years  of  experi¬ 
ence  in  a  field  of  application  for  ana¬ 
lysts.  (See  X^  for  definition  of 
analysts. ) 

Average  number  of  years  of  experi¬ 
ence  in  a  field  of  application  for  pro¬ 
grammers.  (See  for  definition 

of  programmers.  ) 

Average  number  of  years  of  experi¬ 
ence  in  a  field  of  application  for  oper¬ 
ations  personnel.  (See  for  defini¬ 
tion  of  operations  personnel.  ) 
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Symbol  Name 


Definition 


Months  of  machine  The  number  of  calendar  months  be- 
maturity  tween  the  first  delivery  date  for  the 

model  of  base  machine  used  by  the 
ADPS  and  the  date  of  initial  checkout 
of  the  ADPS  computer  programs.  The 
delivery  date  is  given  by  Adams  Asso¬ 
ciates,  Computer  Characteristics 
Quarterly;  the  April  1966  edition  was 
used  for  the  purpose  of  this  study. 
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APPENDIX  D 
COLLECTED  DATA 

(INCLUDING  RELIABILITY  STATEMENTS, 
MEANS,  AND  STANDARD  DEVIATIONS) 
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PHASE  II  COLLECTED  DATA  AND  STATISTICS 
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METHODOLOGY  USED  IN  1  HE  COST  REGRESSION  ANALYSIS 
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A. 


Introduction 


This  appendix  will  set  forth  the  computational  formulas  employed 
in  obtaining  the  quantitative  results  of  the  statistical  analysis.  The  ac¬ 
tual  computations  were  run  on  a  Control  Data  3600  computer.  The  com¬ 
puter  programs  used  to  perform  these  computations  are  part  of  the 
BIOMED  package  programmed  at  the  UCLA  Medical  Center  (see  Refer¬ 
ence  1)  and  are  designated  by  the  three  letters  "BMD"  followed  by  two 
numbers  and  another  letter  (i.  e.  ,  BMD02R). 

B.  Scatterplots 

Scatter  diagrams  were  obtained  by  using  the  plot  option  available 
on  BMD02D,  Correlation  with  Transgeneration.  Plots  are  made  of 
pairs  of  values  (Xij,  X^)  where  the  value  of  Xij  is  plotted  on  the  hor¬ 
izontal  axis  and  the  value  of  is  plotted  on  the  vertical  axis,  and 

where 

j,  k  =  1,  2,  ‘  '  '  ,  p 
i  =  1,  2,  ‘  ,  n 

p  =  number  of  variables 
n  =  number  of  systems 

For  a  single  plot,  j  and  k  are  held  constant,  and  i  varies  1,  2,  •  •  •  ,n  . 

C .  Logarithmic  Transformation 

To  perform  a  logarithmic  transformation,  Xij  is  transformed  to 
the  equivalent  value  in  the  logarithm  to  the  base  10  scale;  in  other  words, 


X.  - 
U 


log ,  X. . 
*10  ij 


X'.. 

u 


Most  BMD  programs  have  a  feature  that  allows  the  user  to  transform 
the  desired  variables  before  the  statistical  analysis  is  performed.  In 
addition,  BMD09S,  Transgeneration,  will  transform  desired  variables 
and  give  as  output  a  card  deck  of  the  transformed  values. 

D.  Correlation 


The  correlation  coefficient,  rjk  ,  as  well  as  the  individual  vari¬ 
able  means  and  standard  deviations,  was  obtained  by  using  BMD03D, 
Correlation  with  Item  Deletion. 

Let  X^  be  the  jth  variable  of  the  ith  system,  where  (i  =  1,  2,  •  ■  •  , 
n)  ,  (k,  j  =  l,Z,---p)  ,  n  is  the  number  of  systems,  and  p  is  the  num¬ 
ber  of  variables.  For  each  Xij  value  that  is  accepted  for  inclusion  in 
the  computation,  the  following  steps  are  performed: 
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1. 


Means 


X  .  = 

•  J 


]_ 

n 


2.  Standard  Deviations 


s 


X.  " 
J 


(X..  -  X  ,)2 
1J  .  J 

n-T 


3.  Correlation  Coefficients 


rjk  = 


Ii<xii-x.i»xik-x.k> 

^i(xij-x./“i(xik-x.k>2 


E.  Analysis  of  Variance 

Analysis  of  variance  techniques  applied  to  test  for  differences 
among  the  means  of  two  or  more  populations  were  t-tests,  one-way 
analysis  of  variance,  and  analysis  of  covariance  (see  References  2  and 

3). 


1.  t-Tests 


t-Tests  are  used  to  test  samples  from  two  populations  to  de¬ 
termine  if  the  means  of  the  two  populations,  |ij  and  jj.2  »  are  equal. 
The  assumption  made  is  that  both  populations  have  normal  distributions 
with  the  same  mean  and  the  same  variance.  Then  the  statistic  tc  has 
a  t(Nl  +  N2  -  2)  distribution.  The  computation  formula  is: 
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where 


„2  (Ni-  DSi+(N2-  1)s| 

p  Nj  +  N?  -  Z 


2.  One-Way  Analysis  of  Variance 

One-way  analysis  of  variance  comprises  tests  for  differ¬ 
ences  among  the  means  of  two  or  more  populations.  In  general,  the  hy¬ 
pothesis  is  m  =  =  •  •  *  »  -  Pk  (the  means  of  all  categories  are  equal). 

The  assumption  is  that  the  observations  are  randomly  selected  from 
normal  populations  with  homogeneous  variance.  Then  the  statistic  Fc 
has  a  distribution  of  F(k-1,  N-k)  .  The  computation  formula  is: 


J  N.(X.  -  X)Z 


N-k 


The  computations  were  made  using  BMD01V,  Analysis  of  Variance*  for 
One-Way  Design. 

3.  Analysis  of  Covariance 

An  analysis  of  covariance  is  performed  by  computer  pro¬ 
gram  BMD04V,  Analysis  of  Covariance  with  Multiple  Covariates.  This 
program  is  designed  to  compute  analysis  of  covariance  information  for 
k  subpopulation  of  Y  values,  where  Y  depends  linearly  on  a  set  of  si¬ 
multaneously  observed  variables,  Xj,  X^,  •  •  •  ,  Xp  .  The  hypothesis  being 
tested  in  analysis  of  covariance  is  stated  as  follows:  There  is  no  differ¬ 
ence  in  the  means  of  the  Y  values  among  groups  after  the  Y  values 
have  been  adjusted  according  to  the  X  values. 

The  analysis  of  covariance  test  assumes  that  the  regression 
curves  in  the  k  populations  are  parallel  straight  lines  and  the  popula¬ 
tion  variances  about  the  regression  lines  are  equal  in  each  of  the  k 
populations . 


119 


F.  Multiple  Regression 


1.  Estimation  Equations 

The  general  linear  model  and  the  requirements  for  estima¬ 
tion  efficiency  were  discussed  in  subsections  II. B. 3  and  4.  The  general 
estimation  equation  is: 


A 

Y.  =  a. 
J  J 


b..X. 
D  1 


A 

where  Y;  will  be  the  predicted  values  for  the  five  cost  variables  and 
Xi  are  the  five  transformed  workload  descriptors  in  logarithms  (base 
10). 


The  stepwise  regression  procedure  BMD02R  (see  Reference  4) 
was  used  because  the  procedure  provides  a  judgment  on  the  contribution 
made  by  each  variable  as  though  it  had  been  the  most  recent  variable  en¬ 
tered.  Variables  incorporated  earlier  are  reexamined  at  every  stage 
of  the  regression.  A  summary  of  the  procedure  is  given  in  the  follow¬ 
ing  paragraphs. 

o  Start  with  the  set  of  causally  related  independent  variables 
and  enter  into  regression  the  Xj  variable  most  highly  cor¬ 
related  with  the  cost  variable  Yj  . 

o  Regress  Yj  on  X^  and  obtain  least- square s  equation. 

Apply  F-test  for  significance. 

°  Calculate  partial  correlation  coefficients  of  all  variables 

not  in  regression  with  cost.  Choose  as  the  next  variable  to 
enter  into  the  regression  the  one  with  the  highest  partial  cor¬ 
relation  coefficient  Xj^  . 

A 

o  Develop  regression  equation  Y  =  f(Xi,  XjJ  by  least  squares. 
Apply  F-test.  Examine  contribution  of  X{  if  X^  had  been 
entered  first.  Apply  F-test  to  X^  and  retain  X^  if  significant. 

o  Repeat  step  3  and  choose  next  variable  Xe  .  Develop  regres¬ 
sion  Y  =  f(X^,  Xj^,  Xe)  by  least  squares. 

Partial  F-tests  are  applied  to  X^,  X^  and,  if  significant,  they  are 
retained  in  the  regression  equation.  If  additional  variables  remain, 
this  procedure  is  continued  until  no  more  variables  will  be  admitted  to 
the  equation  and  no  more  are  rejected. 
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After  each  variable  is  added,  the  following  test  is  performed  until 
the  following  hypothesis  is  rejected:  The  X^^  term  makes  a  signifi¬ 
cant  addition  to  the  regression  equation. 


^Regression  sum  of  squares  for 
:Z' 


Y  =  £(Xl,  X2,...X.,X.  +  1) 


'Regression  sum  of  squares  for'' 
Y  =  f(X1,X2,---X.) 


''Residual  mean  square  for^ 

Y  -  f(X1,X2,-.  •  ,X.) 


where  F  is  an  F-distribution  with  (1,  N-p-1)  degrees  of  freedom;  N 
is  the  number  of  observations  in  the  sample  and  p  is  number  of  work¬ 
load  descriptors  in  the  equation. 

The  result  is  a  vector  of  parameters  a j ,  bl,  b2,  •  •  •  ,  bm  that 
provides  the  best  set  of  estimators  for  aj,  pi,  p2,  •  •  •  ,  Pm  of  the  lin¬ 
ear  regression  model.  In  practice,  it  is  desirable  to  keep  the  number 
of  X^s  in  the  estimating  relationship  with  Yj  small  because  the  user 
prefers  a  minimum  of  effort  in  estimating  cost. 

The  computer  program  BMDOZR,  Stepwise  Regression,  computes 
and  outputs  the  following  statistics  at  each  step: 


o 

o 

o 

o 


o 

2. 


Multiple  correlation  coefficient,  R 

Standard  error  of  estimate  of  Y .  ,  s. 

J  J 

Analysis  of  variance  table 
For  variables  in  the  equation: 

a.  Regression  coefficient 

b.  Standard  error  of  regression  coefficient 

For  variables  not  in  the  equation,  partial  correlation  coefficient 
Reliability 


Total  variance  pertains  to  the  deviations  of  the  sample  Y1  s 
from  their  mean.  *  Explained  variance  refers  to  the  deviations  from  Yj 
of  the  computed  Yj  values  (calculated  from  the  regression  equation) 
corresponding  to  the  values  of  in  the  sample.  Unexplained  variance 

is  derived  from  tjie  deviations  of  the  sample  Yj  values  from  the  com¬ 
puted  values  of  Y .  . 
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(T 


tj 


cr  .  +  cr  . 
eJ  uj 


Y 


a,  Coefficient  of  Variation 

This  is  a  relative  measure  for  standard  error  of  esti¬ 
mate,  which  is  the  square  root  of  the  unexplained  variance  adjusted  by 
the  number  of  degrees  of  freedom  (see  Reference  5). 


C  = 


degrees  of  freedom 


Y. 

J 


Typically,  0  <:  C  £  0,2  is  desirable, 

b.  Coefficient  of  Correlation 


The  correlation  coefficient  is  a  measure  of  the  degree 
of  association  between  the  dependent  variable  and  the  explanatory  vari¬ 
ables,  R  is  defined  as  the  square  root  of  the  proportion  of  total  variance 
that  is  represented  by  the  explained  variance. 


2 


2 


Typically,  1.0  ^  R  ^  0.8  is  desirable. 
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c.  Residual  Analysis 

A 

Analysis  of  residuals,  the  unexplained  deviations  Yj-Yj, 
is  a  useful  tool  for  further  analysis  of  the  unexplained  variance.  In 
addition,  it  provides  analytic  tests  for  truths  of  assumptions  made  in 
regression  analysis.  These  are  that  errors  or  deviations  are  independ¬ 
ent,  have  zero  mean  and  a  constant  variance  c r^,  and  are  normally  dis¬ 
tributed,  The  usual  procedure  is  to  plot  the  residuals  as  shown  in 
Figure  15  (see  Reference  6). 

3.  Prediction  Intervals 


The  preceding  measures  are  measures  of  reliability  con¬ 
sidered  in  the  context  of  the  regression  equation  in  relation  to  the  sam¬ 
ple  observations.  As  a  measure  of  predictive  efficiency,  the  concept 
of  the  prediction  interval  is  used.  For  given  values  of  the  explanatory 
variables  X—  's,  the  estimating  equation  is  used  to  obtain  a  predicted 
value  Yj.  A  boundary  is  placed  around  Y j ,  Yj  ±  A,  such  that  there  is 
a  certain  level  of  confidence  that  the  established  interval  brackets  the 
population  value  of  Yj.  An  80  percent  prediction  interval  does  not  mean 
that  the  probability  is  0.80  that  the  population  value  of  Yj  lies  between 
that  interval.  Rather  it  means  that  there  is  80  percent  confidence,  in 
a  subjective  sense,  that  this  is  the  case.  This  is  fiducial  probability 
and  not  a  true  probability  statement. 

In  the  case  of  one  explanatory  variable,  this  interval  typically 
can  be  depicted  as  the  area  between  two  hyperbolae,  one  on  each  side 
of  the  regression  line.  In  the  case  of  two  explanatory  variables,  this 
interval  can  be  depicted  as  the  space  between  two  hyperboloids,  one  on 
each  side  of  the  regression  plane.  For  larger  numbers  of  explanatory 
variables,  this  interval  cannot  be  graphically  portrayed,  and  the  equa¬ 
tion  for  the  interval  becomes  increasingly  more  complicated  (see 
Reference  7). 

The  1-a  prediction  limits  for  a  Yj  obtained  from  a  particular 
set  of  values  are  given  by 
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G.  Factor  Analysis 

The  factor  analysis  was  performed  by  using  BMD03M,  General 
Factor  Analysis.  This  program  uses  a  principle  component  method 
with  an  orthogonal  rotation  of  the  factor  matrix.  Communalitie s  are 
estimated  from  the  squared  multiple  correlation  coefficients.  The 
complete  computational  procedure  is  given  in  BMD03M.  For  further 
details  on  factor  analysis  see  Reference  8. 

H.  Hardware  Costs 


The  hardware  costs  for  the  18  systems  surveyed  were  calculated 
by  using  the  following  computation  procedure: 

1.  For  program  checkout 


Let 


A 

S 


=  total  checkout  hours  for  subject  ADPS,  base 
machine 

=  total  checkout  hours  for  subject  ADPS,  satellite 
machine 

=  basic  hourly  rental,  base  machine^ 

=  basic  hourly  rental,  satellite  machine^ 

=  dollars  of  hardware  cost  for  program  checkout 


For  purchased  components,  the  applicable  GSA  monthly  rental  costs 
were  used. 


125 


Then 


Y2  =  ARb  +  SRg 

2.  For  application  production  or  application  program  maintenance 
Let  A  =  monthly  hours  for  AD  PS,  base  machine 
B  =  total  monthly  hours,  base  machine 
S  =  total  monthly  hours,  satellite  machine 


Rg  =  basic  hourly  rental,  base  machine 


(1) 


Rg  =  basic  hourly  rental,  satellite  machine 

Cg  =  central  processing  unit  extra  time  hourly  rate, 
base  machine 

Cg  =  central  processing  unit  extra  time  hourly  rate, 
satellite  machine 


Where 

Yp-  =  dollars  per  month  of  hardware  cost  for  application 
production 

=  dollars  per  month  of  hardware  cost  for  program 
maintenance 

Case  I:  A  <  200,  AS/B  <  200;  then 


Y5’Y6=  KJ  +  [¥  Rsl 


Case  II:  A  <  200,  AS/B  >  200;  then 


+  200  R, 


20o)  C< 


Case  III:  A  >  200,  AS/B  <  200;  then 


Y5'Y6=  I200  RB  +  (A-200>  CbHtT  Rs] 


For  purchased  components,  the  applicable  GSA  monthly  rental  costs 
were  used. 
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Case  IV:  A>  200,  AS/B  >  200;  then 


Y5Y6  =  200  RB 


+  (A  -  200)  C  J  + 

JdJ 


200  Rs  +  (¥  -  20°)  Cs  I 


Note  that  the  preceding  procedure  is  applicable  for  an  instal¬ 
lation  with  at  most  one  base  and  one  satellite  computer.  If 
there  are  two  base  or  satellite  computers  and: 


Case  V:  A  <  400,  AS/B  <  400;  then  use  the  appropriate 
Cases  I  through  IV  without  making  any  changes. 

Case  VI:  Either  A  >  400  and  two  base  computers  are  in¬ 
stalled  or  AS/B  >  400  and  two  satellite  computers  are  in¬ 
stalled,  or  both;  then  use  the  appropriate  Cases  I  through 
IV,  and,  where  there  are  two  computers,  change  the  con¬ 
stants  200  to  400  as  appropriate. 
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APPENDIX  F 

CURRENT  ADPS  PROPOSAL  PROCEDURES 


13  1 


A. 


Introduction 


One  of  the  major  objectives  of  this  contract  is  to  propose  tools  to 
the  decision  makers  at  HQ  USAF  to  assist  them  in  judging  proposals  for 
new  automation.  For  any  tool  to  be  constructed  in  the  most  useful  man¬ 
ner,  it  is  necessary  to  understand  who  the  decision  makers  are,  what 
analytical  procedures  they  follow  in  judging  proposals  fur  new  automa¬ 
tion,  and  what  the  form  and  content  of  such  proposals  are.  To  the  extent 
possible  within  contract  scope,  the  PRC  project  team  has  gathered  such 
data  through  a  study  of  applicable  Air  Force  regulations  and  through  many 
lengthy  discussions  with  personnel  at  HQ  USAF. 

This  appendix  summarizes  the  various  regulatory  procedures  that 
govern  the  preparation  and  submission  of  proposals  involving  ADP  sys¬ 
tems  to  HQ  USAF.  It  is  not  claimed  that  these  represent  all  applicable 
procedures,  but  PRC  is  certain  that  the  majority  of  all  ADPS  proposals 
are  covered  by  the  regulations  discussed  herein.  It  should  be  clear,  after 
perusal  of  this  appendix,  just  how  complex  the  proposal-judging  function 
is  and  how  urgently  the  decision  makers  need  additional  tools. 

Specifically,  the  remainder  of  this  appendix  discusses  300  series 
regulations  and  the  functions  of  AFADA,  375  and  57  series  regulations 
and  system  management  procedures,  100  series  regulations  governing 
communications  systems,  and  AFR  80-2  concerning  research  and 
development . 

Various  organizations  within  the  Air  Force  are  referenced  herein 
and  the  organization  chart  presented  in  Figure  16  should  help  identify 
the  position  of  a  given  organization  within  the  Air  Force  structure. 

R.  AFR  300  Series  Regulations 

This  series  deals  in  general  with  the  design,  implementation,  and 
operation  of  automated  data  systems  for  management  supporting  data  sys¬ 
tems,  operations  supporting  systems,  and  research  and  development  sup¬ 
porting  data  systems.  It  also  pertains  to  the  selection,  acquisition,  and 
management  of  automatic  data  processing  equipment  for  these  systems, 
with  the  following  notable  exceptions: 

o  Data  systems  and/or  equipment  integral  to  a  weapon  system 

o  ADPS  under  development  for  a  particular  use  through  the 

expenditure  of  research  and  development  test  and  evaluation 
funds 

o  Analog  computing  systems 

AFR  300-2  establishes  the  Air  Force  general  objectives  and  policies 
in  the  area  of  data  automation  and  specifies  that  the  Senior  ADP  Policy 
Official  for  the  Air  Force  is  the  Assistant  Secretary  of  the  Air  Force 
(Financial  Management).  In  this  capacity,  he  is  responsible  for  the 
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administration  of  the  Air  Force  ADP  program  and  the  selection  and  acquisi¬ 
tion  of  ADP  equipment;  accordingly,  all  proposals  for  ADP  equipment  acqui¬ 
sition  must  be  approved  by  him.  AFADA  has  been  designated  by  SAFFM 
as  the  focal  point  for  coordinating  and  integrating  the  Air  Force  data  auto¬ 
mation  effort.  Functions  performed  by  AFADA  will  be  covered  in  subse¬ 
quent  paragraphs. 

1 .  AFR  300-3,  Management  Supporting  Data  Systems 

This  regulation  establishes  procedures  and  responsibilities 
for  the  design,  implementation,  modification,  and  maintenance  of  man¬ 
agement  supporting  data  systems.  In  most  cases  a  Data  Automation 
Proposal  (DAP)  is  mandatory.  Procedures  and  formats  for  DAP  prepa¬ 
ration  and  submission  are  included  in  this  regulation.  Program  control 
of  design  and  implementation  of  management  supporting  data  systems  is 
exercised  through  the  Data  System  Automation  Program  (DSAP).  HQ 
USAF  makes  DSAP  entries,  reflecting  the  separate  design  and  implemen¬ 
tation  phases  of  automated  data  systems,  as  follows: 

o  Systems  Development  Projects  Inventory.  This  entry  re¬ 
flects  issuance  of  a  Data  Project  Directive  and  indicates 
data  system  design  activity  by  location  and  scheduled  com¬ 
pletion  date. 

o  Data  System  Implementation  Schedule.  This  entry  reflects 

current  implementation  plans  and  identification  of  the  support 
ADP  equipment  scheduled  for  each  location. 

o  Current  System  Inventory.  This  entry  reflects  current  active 
data  systems  and  ADP  equipment  in  use  in  support  of  such 
data  systems. 

Reporting  procedures  are  those  outlined  in  AFM  171-9. 

Systems  proposed  under  this  regulation  are  categorized  as  either 
standard  or  unique.  Standard  data  systems  are  common  to  two  or  more 
commands  or  agencies  and  possess  uniformity  of  inputs,  file  content, 
processing  logic,  and  outputs.  Unique  data  systems  are  peculiar  to  a 
single  command  or  agency. 

HQ  USAF  (AFADAC)  must  review  DAP's  received  to  determine  the 
following: 

o  Acceptance,  and  (a)  establishment  of  a  system  development 

project,  (b)  other  directed  action  prior  to  implementation,  or 
(c)  directed  implementation 

o  Nonacceptance,  and  (a)  return  for  additional  information  or 

development,  or  (b)  return  with  explanation  of  nonacceptability 
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FIGURE  16  -  AIR  FORCE  ORGANIZATION  CHART  (PARTIAL) 


Because  AFADA  is  the  decision  authority  for  management,  opera¬ 
tions,  and  research  and  development  supporting  data  systems,  something 
should  be  said  at  this  point  concerning  its  organization,  functions,  and 
overall  responsibilities.  All  of  these  are  covered  in  detail  in  AFM  170-6; 
however,  it  should  prove  instructive  to  describe  those  functions  associated 
with  the  approval  process  for  DAP’s. 

Figure  17  shows  the  organization  of  AFADA.  All  DAP's  go  to 
AFADACA  for  coordination  and  evaluation.  It  is  their  responsibility  to 
see  that  all  interested  members  of  the  Air  Staff  are  involved  in  the  eval¬ 
uation  process.  Each  DAP  is  logged  in  and  given  a  number.  The  goal  at 
AFADACA  is  to  completely  process  a  DAP  in  no  longer  than  45  days. 

The  DAP  is  subjected  simultaneously  to  an  in-house  review  and  a  func¬ 
tional  review.  The  functional  review  consists  of  sending  the  DAP  to  any 
part  of  the  Air  Staff  which  might  be  involved  or  interested  (e.g.  ,  DCS/ 
Personnel  if  additional  manpower  is  required). 

The  in-house  review  consists  of  sending  the  DAP  to  those  parts  of 
AFADA  which  might  have  some  comment,  and  almost  always  includes 
AFADAA,  AFADAB,  AFADAE,  and  AFADO.  Typical  responsibilities 
of  these  organizations  are  as  follows: 

1.  AFADAA .  Key,  but  not  all  inclusive,  responsibilities  as 
described  in  AFM  170-6  are: 

"Reviews,  validates,  and  has  approval  authority  for  all 
data  system  content  and  standard  output  therefrom  (AFR300 
series).  Insures  standardization  of  this  data  to  provide  in¬ 
terface  capabilities  and  to  preclude  non-essential  overlap  or 
duplication  within  and  between  systems  and  reports. 

"Prescribes  the  system  and  procedures  for  a  continu¬ 
ous  Air  Force-wide  review,  analysis  and  validation  of 
all  reports,  data  bank  content,  and  standard  outputs. 
Conducts  periodic  reviews  of  all  reporting  requirements 
placed  on  the  Air  Force  by  other  Federal  agencies  and 
the  public. 

"Directs  and  is  responsible  for  the  Air  Force  Data  Ele¬ 
ments  and  Codes  Standardization  program  including  the 
approval,  publication  and  implementation  of  standard 
data  elements,  data  items,  data  codes,  data  descriptors, 
and  data  field  designators.  Provides  guidance  and  ad¬ 
vice  to  Data  Automation  Working  Groups  on  these  mat¬ 
ters.  Resolves  functional  area  conflicts. 

"Establishes  and  controls  automated  file(s)  for  data 
elements  and  related  features  (data  items,  codes,  de¬ 
scriptors,  and  field  designators),  including  a  repository 
of  the  data  content  of  standard  data  banks  and  Headquar¬ 
ters  USAF  directed  or  implemented  reports. 
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FIGURE  17  -  ORGANIZATION  OF  AFADA 


"Evaluates  information  requirements  of  the  Secretary 
of  the  Air  Force,  Chief  of  Staff,  and  other  principal 
Air  Staff  officers.  Assures  that  valid  requirements 
are  in  data  banks  or  reports.  ” 

Accordingly,  AFADAA's  main  function  with  respect  to  DAP 
review  is  to  insure  that  reports,  data  elements,  codes,  etc.  , 
are  in  compliance  with  AFR  174-1  and  AFR  300-4  as  required. 

2.  AFADAB .  Again  quoting  from  AFM  170-6,  key  responsibili¬ 
ties  of  this  organization  include: 

"Serves  as  focal  point  and  is  responsible  for  data  auto¬ 
mation  objectives,  concepts,  plans  and  policies  in  sup¬ 
port  of  overall  Air  Force  objectives  and  plans. 

’’Develops  the  regulatory  structure  for  effective  manage¬ 
ment  of  the  total  data  automation  effort. 

’’Serves  as  the  Air  Force  focal  point  with  DOD  on  all 
matters  pertaining  to  data  automation  objectives,  con¬ 
cepts  and  policies,  and  as  the  AFADA  coordinating 
office  on  all  DOD  matters. 

’’Establishes  and  coordinates  Air  Force  requirements 
for  technical  data  automation  studies  and  development 
projects;  monitors  their  progress  and  evaluates  results. 

’’Establishes  policies  pertaining  to  data  automation  tech¬ 
nical  standards  for  Air  Force  use,  and  coordinates  the 
development  and  adoption  of  technical  standards  with 
other  agencies  or  industry. 

’’Plans  for  the  interface  and  integration  of  Air  Force 
management  and  operational  supporting  data  systems 
to  insure  efficiency  and  elimination  of  duplication.  ’’ 

In  reviewing  a  DAP,  AFADAB  determines  whether  regulations 
in  addition  to  the  AFR  300  series  should  apply  and  whether  es¬ 
tablished  standards  are  involved  or  suggested. 

3.  AFADAE.  Key  functions  as  stated  in  AFM  170-6  include: 

"Exercises  surveillance  over  USAF  data  automation 
installations;  evaluates  progress  and  performance 
against  programs  and  standards;  and  initiates  correc¬ 
tive  action  when  necessary. 

"Plans  for  and  monitors  the  installation,  operation,  and 
management  of  all  ADP  Equipment  after  the  equipment 
selection  and  approval  process  has  been  completed. 
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"Prescribes  and  manages  the  USAF  Data  Systems  Auto¬ 
mation  Program  (DSAP)  and  changes  thereto. 

"Reviews  requests  for  ADPE  and  recommends  approval 
action  based  on  budget  requirements  and  current  man¬ 
agement  actions. 

"Reviews  and  approves  requests  for  ADP  services 
through  service  contracts. 

"Compiles  Data  Automation  program  cost,  ADPE  util¬ 
ization  and  inventory  data  for  the  Air  Staff,  OSD,  BOB 
and  other  Government  agencies  use. 

"Performs  continuous  post  installation  studies  of 
method  of  acquisition  of  ADPE  and  initiates  purchase 
action  when  economically  advantageous. 

"Administers  the  relocation  or  disposition  of  surplus 
Government-owned  ADP  Equipment." 

Manpower  implications  in  the  DAP  are  analyzed  and  discussed. 

4.  AFADO.  This  organization  determines  whether  the  system 
proposed  in  the  DAP  is  unique  or  standard.  It  might  also 
recommend  holding  up  a  proposed  unique  system  because  of 
some  standard  system  already  under  development.  If  a  pro¬ 
posed  unique  system  has  Air  Force-wide  benefits,  AFADO 
might  establish  it  as  a  standard  system.  AFADO  maintains 
the  Air  Force's  standard  Management  Supporting  Data  Sys¬ 
tems  and  normally  implements  such  systems. 

The  instructions  for  preparing  a  DAP  are  included  as  Attachment  2 
of  AFR  300-3.  A  copy  of  this  attachment  is  presented  in  Figure  18.  The 
current  instructions  call  for  only  additional  resources  required.  Current 
practice  at  AFADAC  is  to  request  all  resources  required  before  a  DAP 
can  be  properly  evaluated. 

Several  key  questions  must  be  answered  when  evaluating  a  DAP, 
all  of  which  are  answered,  with  varying  degrees  of  success,  by  AFADAC 
proposal  evaluators: 

o  Does  the  Air  Force  need  it?  In  other  words,  does  the  pro¬ 
posed  ADPS  fall  within  the  policies  and  objectives  of  the  Air 
Force  as  a  whole  and  the  specific  mission  of  the  requestor? 
This  is  by  far  the  hardest  question  to  answer  and,  once  an¬ 
swered,  the  one  most  subject  to  argument. 

o  If  a  valid  mission  requirement  exists,  is  the  proposed  ADPS 
the  best  technical  and  most  economical  solution?  And,  as  a 
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AFR  300-3 


DATA  AUTOMATION  PROPOSAL  (DAP)  SUBMISSION 


General  Instructions*  Complete  detail  pertaining  to  each  DAP  item  may  not  be  available  (or  re¬ 
quired)  at  the  time  of  DAP  submission.  However,  each  item  should  be  completed  to  the  degree  appro¬ 
priate  at  the  time  of  submission.  Items  not  directly  pertinent  to  the  specific  proposal  should  be  marked 
“Not  Applicable/1  The  following  format  must  be  followed: 

1.  Identification.  Indicate  originating  base  and/or  organization,  parent  command,  and  prepara¬ 
tion  date. 

2.  Title  and  Purpose.  Identify  the  data  automation  requirement/recommendation;  specify  what 
is  to  be  accomplished;  and  relate  this  to  an  established  function  or  responsibility;  specify  the  data  auto¬ 
mation  characteristics  involved;  and  indicate  any  associated  organizational  and  procedural  changes 
contemplated. 

3.  Syslcin/Modificalion  Description.  Specify  the  inputs  and  file  content,  and  provide  a  general 
How  diagram  showing  processing  operation.  Identify  outputs  and  their  relationship  with  other  data 
systems.  Indicate  processing  workload,  responsiveness  criteria,  etc.,  at  appropriate  points  within  the 
processing  operation. 

1.  Resource  Requirements.  Indicate,  to  the  degree  possible,  the  anticipated  additional  resources 
required  (over  those  now  in  use)  for  the  proposed  system  or  modification  under  normal  operating  con¬ 
ditions.  Resource  requirements  should  be  specified  as  being  command  or  Air  Force-wide,  separately 
identified  within  the  following  groups: 

a.  Personnel  (grade/man  months  or  years;. 

b.  Equipment  (identify,  and  include  approximate  dollar  cost). 

c.  Physical  facilities  (site  preparation,  approximate  dollar  cost). 

d.  Communications  (identify  number  of  units,  approximate  dollar  cost). 

e.  Other  (as  appropriate). 

5.  Summary  of  Benefits.  Indicate,  to  the  degree  practicable,  the  economics  and/or  other  benefits 
to  accrue  on  a  command  or  Air  Force-wide  basis  through  the  proposed  system  or  modification.  Tangible 
benefits  (personnel,  equipment,  or  other  savings)  should  be  summarized  to  indicate  an  estimated  dollar 
value  for  a  specific  time  period.  Intangible  benefits  (increased  efficiency  or  responsiveness,  accomplish¬ 
ment  of  tasks  not  previously  feasible  or  possible,  preclusion  of  increased  cost  of  current  operations, 
etc.)  should  be  outlined  in  narrative  form,  with  explanation  of  derivation  of  the  benefit. 

6.  Remarks.  Include  additional  information  which  would  facilitate  understanding  and  evaluation 
of  the  submitted  DAP.  For  new  Unique  Data  Systems  include  a  schedule  of  proposed  locations,  if 
applicable. 


FIGURE  18  -  PRESCRIBED  FORMAT  FOR  DATA  AUTOMATION  PROPOSALS 
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corollary  to  this  question,  is  there  an  existing  Air  Force 
ADPS  that  will  do  the  job,  or  do  other  ADPS  proposals  in 
process  support  or  conflict  with  the  subject  proposal? 

It  is  in  answering  these  questions  that  better  tools  would  be  most 
useful  to  the  proposal  evaluators.  Although  they  are  currently  doing  an 
adequate  job  in  this  area,  they  are  not  equipped  to  contend  with  increases 
in  the  proposal  load  and  continuing  expansion  of  data  processing  in  the 
Air  Force;  current  procedures  will  become  increasingly  prone  to  error, 
and  the  time  to  process  a  proposal  will  become  longer  and  longer.  More 
than  700  DAP's  have  been  processed  by  HQ  USAF  in  the  last  5  years;  of 
these,  over  half  were  submitted  within  the  last  12  months.  If  the  load 
continues  to  increase  at  this  rate,  better  tools  and  procedures  are 
mandatory. 

At  present,  the  tools  available  to  proposal  evaluators  are  essen¬ 
tially  a  listing  of  past  and  current  DAP's  in  numerical  order  and  the  Data 
System  Automation  Program  (DSAP).  The  officers  within  AFADAC  who 
perform  proposal  evaluations  have  functional  areas  of  responsibility, 
which  minimizes  the  amount  of  information  with  which  they  must  become 
familiar  and  remember.  However,  these  procedures  can  accommodate  an 
increased  workload  only  by  adding  more  people  and  establishing  a  finer 
functional  stratification.  Furthermore,  there  are  at  present  no  tools, 
except  the  experience  of  the  individual  officers  performing  the  evaluation, 
for  assessing  cost  estimates. 

Other  responsibilities  of  AFADA  covered  by  this  regulation  deal 
with  procedures  to  be  followed  after  a  DAP  is  approved. 

In  many  cases  it  is  deemed  desirable  to  establish  a  system  devel¬ 
opment  project  for  the  design  (or  modification)  of  automated  data  systems, 
development  of  associated  data  system  specifications,  and  demonstration 
of  the  operational  feasibility  of  new  concepts  and  techniques.  In  this 
event,  a  Data  Project  Directive  (DPD)  is  issued  by  AFADA  which  pro¬ 
vides  the  charter  for  command  or  agency  initiation  of  a  system  develop¬ 
ment  project.  One  of  the  key  documents  produced  by  the  system  develop¬ 
ment  project  is  the  Data  System  Specifications,  which  provide  a  complete 
description  of  the  specific  system,  including  identification  of  related 
standard  data  systems,  pertinent  standard  data  elements  and  codes,  input 
and  output  definitions,  file  and  record  content,  and  logical  flow  diagrams 
of  the  functions  performed.  If  the  Data  System  Specifications  are  approved 
by  HQ  USAF,  an  implementation  schedule  is  prepared  and  sent  to  the  com¬ 
mand  or  agency,  which  in  turn  prepares  the  following: 

o  Available  ADP  equipment  capability 

o  Funding  requirements 

o  Workload  confirmation 
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o  Site  preparation  requirements 

o  Training  requirements 

o  Verification  of  benefits 

When  all  approvals  have  been  made,  a  final  implementation  plan  is 

developed  to  ensure  orderly  and  effective  implementation  of  the  data 
system. 

2.  Operations  Supporting  Data  Systems 

ADP  systems  for  operations  supporting  data  systems  currently 
are  acquired  through  AFR  300-3  (DAP’s)  or  AFR  375-1  (ROC's).  A  draft 
version  of  AFR  300-6,  which  covers  this  area,  is  being  studied  by  AFADA; 
if  adopted,  these  systems  will  receive  uniform  treatment. 

3.  AFR  300-7,  Research  and  Development  Supporting  Systems 

This  regulation  distinguishes  between  research  and  develop¬ 
ment  support  and  management  or  operational  supporting  data  systems. 

It  prescribes  responsibilities  for  establishing  and  providing  scientific/ 
computational  ADP  equipment  support  required  in  conjunction  with  ap¬ 
proved  research  and  development  activity.  Requirements  for  new  or  ad¬ 
ditional  ADP  equipment  needed  primarily  to  support  administration  and 
management  of  research  and  development  programs  must  be  initiated  and 
developed  in  accordance  with  AFR  300-3. 

Requests  are  submitted  to  AFADAC  in  the  form  of  a  letter  of  trans¬ 
mittal.  If  new  equipment  is  required,  an  equipment  specification  must 
be  attached  to  the  letter  of  transmittal.  The  letter  must  include  the 
following: 

o  A  statement  explaining  why  augmentation  of  existing  ADP 
equipment  cannot  satisfy  the  requirement 

°  An  analysis  of  the  feasibility  of  sharing  equipment  with  other 
Air  Force  or  Government  agencies 

°  Justification  for  special  equipment  features,  etc. 

o  A  description  of  the  tasks  and  their  associated  workload  (ma¬ 
chine  hours  and  additional  manpower) 

Although  format  requirements  are  different  from  a  DAP,  the  infor¬ 
mation  required  is  similar.  AFADA  actions  are  also  similar.  They  in¬ 
clude  the  following: 

o  Review  and  evaluate  the  requests 
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o  Screen  requirements  for  possible  reutilization  of  available 
excess  Government-owned  or  -leased  ADP  equipment 

°  Forward  equipment  specifications  to  ESD,  AFSC,  for  initia¬ 
tion  of  ADP  equipment  selection  process 

o  Obtain  higher  authority  approval  for  waiver  of  competitive 
ADP  equipment  selection,  when  required 

°  Advise  the  major  air  command  to  initiate  appropriate  ADP 
equipment  acquisition  action 

4.  HOI  300-3,  Management  Supporting  Data  Systems 

This  supplements  AFR  300-3  and  establishes  Air  Staff  respon¬ 
sibilities  in  accord  with  DOD  Directives  4105.55  and  5100.40.  Key  func¬ 
tions  of  AFADA  outlined  in  this  document  are  as  follows: 

o  Develop  and  maintain  a  data  system  designator  (short  title) 
system  for  data  system  identification 

o  Ensure  standardization  and  avoid  non-essential  overlap  and 
duplication  of  data  systems 

o  Prescribe  standard  machine  programming  language(s)  to  be 
used 

o  Maintain  and  publish  the  USAF  DSAP 

o  Disseminate  periodically  status  of  DAP's,  DPD's,  and  re¬ 
lated  actions 

o  Maintain  and  prepare  AFM  300-4,  all  approved  standard 
data  elements  and  codes 

C.  AFR  37  5  and  57  Series  Regulations 

System  management  in  the  Air  Force  is  defined  as  the  process  of 
planning,  organizing,  coordinating,  evaluating,  controlling,  and  direct¬ 
ing  the  combined  effort  of  Air  Force  contractors  and  participating  orga¬ 
nizations  to  accomplish  system  program  objectives.  The  documents  of 
primary  interest  are  AFR  375-1  and  HOI  375-1,  Management  of  System 
Programs. 

Programs  that  come  under  this  type  of  management  are  defined  as 
follows : 

1.  Mandatory.  All  new  (or  major  modifications  of  existing)  pro¬ 
duction  systems,  or  new  engineering  and  operational  systems 
developments  shall  be  managed  according  to  AFR  375-1  and 
HOI  37  5-1  if  they  fulfill  one  or  both  of  the  following  stipulations 
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a.  The  program  is  rated  in  the  BRICK-BAT  category 
(AFR  70-24). 

b.  The  program  is  estimated  to  require  total  cumulative 
RDT&E  financing  in  excess  of  $25  million;  or  estimated 
to  require  a  total  production  investment  in  excess  of 

$  1  00  million. 

2  Otherwise  Designated.  Other  system  programs  may  be  des¬ 
ignated  for  this  type  of  management  when  they  possess  one  or 

more  of  the  following  characteristics: 

a.  The  program  significantly  affects  U.  S.  military  posture. 

b.  The  program  is  closely  related  and,  when  taken  collec¬ 
tively,  would  qualify  under  dollar  thresholds  given  above. 

c.  Significant  technical  problems  are  anticipated. 

d.  Unusual  organizational  complexity  or  technological 
advancement  is  involved. 

e.  Extensive  interdepartmental,  national,  or  international 
coordination  or  support  is  required. 

f.  Technological  risks  are  involved  that  may  cause  diffi¬ 
culties  in  many  functional  areas. 

g.  Unusual  difficulties  are  presented  that  require  expedi¬ 
tious  handling  to  satisfy  an  urgent  requirement. 

In  general,  the  purpose  of  applying  systems  management  is  to  en¬ 
sure  that  efforts  by  functional  activities  of  the  Air  Force  are  accomplished 
consistent  with  the  objectives  of  each  system  program.  Complexity,  long 
lead  time,  extensive  resource  requirements,  and  urgent  necessity  to  at¬ 
tain  and  maintain  maximum  operational  capability  are  factors  that  make 
it  mandatory  to  apply  system  management  procedures. 

Until  recently,  a  system  project  of  the  type  discussed  started  when 
a  QOR  (Qualitative  Operational  Requirement),  SOR  (Specific  Operational 
Requirement),  OSR  (Operational  Support  Requirement),  or  ADO  (Advanced 
Development  Objective)  was  written.  AFR  57-1,  17  June  1966,  establishes 
the  ROC  (Required  Operational  Capability)  as  the  replacement  for  QOR's, 
and  the  RAD  (Requirements  Action  Directive)  as  the  replacement  for  SOR's, 
OSR's,  and  ADO's. 

The  ROC  is  a  command's  official  request  to  HQ  USAF  for  a  new  or 
improved  operational  capability  and,  although  any  organizational  level 
may  originate  such  a  document,  it  must  be  signed  by  a  general  officer 
or  a  colonel  occupying  a  key  staff  position. 
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The  RAD  is  prepared  by  HQ  USAF,  signed  by  a  general  officer  at 
directorate  level;  it  directs  and  guides  the  Air  Force  actions  necessary 
to  translate  a  required  operational  capability  into  an  approved  and  funded 
program.  The  RAD  is  a  guidance  document,  not  a  funding  instrument; 
however,  it  transmits  the  funding  information  available  at  the  time  it  is 
issued. 

The  focal  point  within  HQ  USAF  for  the  coordination  of  ROC  proc¬ 
essing  is  AFRDQ.  Key  functions  performed  include  the  following: 

o  Evaluate  the  requirement  and  initiate  actions  to  include,  but 
not  be  limited  to,  such  items  as: 

a.  Preparing  a  plan  of  action  to  evaluate  the  need  and 
satisfy  or  to  disapprove  the  requirement 

b.  Initiating  and  conducting  further  studies  involving  sys¬ 
tem  analysis,  tradeoffs,  cost  effectiveness,  etc. 

c.  Directing  and  guiding  actions  required  of  AFSC,  AFLC, 
and  other  major  air  commands  through  the  RAD 

o  Evaluate  proposed  technical  approaches  submitted  by  AFSC, 
AFLC,  industry  sources,  and  other  commands. 

o  Determine  the  best  acceptable  approach,  with  participation 

of  others  as  necessary,  and  submit  a  proposal  to  appropriate 
levels  of  approving  authority.  An  RAD  is  normally  issued 
within  60  days  of  receipt  of  an  ROC. 

o  Resolve  requirements  with  allied  nations  and  achieve  inter¬ 
service  coordination  as  required. 

Once  a  system  project  is  established  under  AFR  375-1,  AFSPDO 
becomes  the  office  of  primary  responsibility  (OPR)  for  establishing  pol¬ 
icy  and  coordinating  activities  within  the  Air  Staff  pertaining  to  system 
program  documentation  and  its  application  to  system  programs.  It  is 
possible  for  a  system  to  have  four  phases:  conceptual,  definition,  ac¬ 
quisition,  and  operational.  The  HQ  USAF  OPR  for  system  program  man¬ 
agement  will,  through  the  system  life  cycle,  be  transferred  to  the  next 
deputate  having  prime  responsibility.  Some  of  the  major  steps  involved 
in  most  system  programs  are  shown  in  Table  8.  Key  documents  in¬ 
volved  in  the  system  life  cycle  are  described  in  the  following  paragraphs. 

1 .  System  Management  Directives  (SMD's) 

These  directives  provide  uniform  HQ  USAF  direction  for  initi¬ 
ating,  changing,  and  terminating  system  programs  under  AFR  37  5-1.  The 
first  SMD  establishes  the  charter  for  conducting  a  system  program  and  will 
designate  application  of  system  management,  transmit  or  reference  the 
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TABLE  8  -  HQ  USAF  SYSTEM  PROGRAM  RESPONSIBILITY 


_ System  Life  Cycle _  Deputy  Chief  of  Staff  OPR 

Conceptual  phase  (concept  formulation) 

Initial  SMD  (charter) 

PTDPj  review- -PCP  processing  AFRDC  (R&D)  or  AFSDC  (S&L) 

PTDP^  review 

Memorandum  or  PCP  processing 
Definition  phase  (contract  definition) 

SMD  issued 
PA  issued 

Budget  authority  issued  by  AFABF 

(Di  rector  of  Budget)  X 

FTA  issued  AFRDC  or  AFSDC 

Contractor  selection 

Memorandum  or  PCP  processing 

PSPP 

Acquisition  phase 
SMD  issued 
SPP  review 
Contracting 
Development  effort 
Production 
PCP/PA/BA 
Category  I,  II  tests 
Updating  changes 
Last  article  delivered 
Transition  agreement 
SMD  issued 
Operational  phase 


AFSDC 


AFXOP  or  other 
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current  requirements  document,  and  request  a  Program  Change  Proposal 
(PCP)  and  either  a  Preliminary  Technical  Development  Plan  (PTDP)  or  a 
Proposed  System  Package  Plan  (PSPP).  If  a  formal  definition  phase  is 
not  planned,  a  PSPP  is  requested  from  the  implementing  command,  not 
a  PTDP.  Although  an  SMD  reflects  policy  decisions  made  within  OSD  and 
HQ  USAF,  including  changes  in  the  Force  and  Financial  Plan  (F&FP),  an 
SMD  in  itself  does  not  constitute  authority  to  let  a  contract.  An  approved 
(signed)  secretarial  Determinations  and  Findings  (D&F)  is  required  be¬ 
fore  contract  negotiations  can  be  initiated  or  an  RFP  issued.  Fund  avail¬ 
ability  is  established  and  a  secretarial  statement  of  Final  Technical  Ap¬ 
proval  (FTA)  is  obtained  before  a  contract  containing  RDT&E  funds  may 
be  signed.  Separate  program  authorizations  (PArs)  issued  by  AFRRP 
(Assistant  for  R&D  Programming)  and  Procurement  Authorizations  (PA's) 
issued  by  AFSPD  provide  procurement  authorization. 

2.  Program  Change  Proposal  (PCP) 

This  document,  submitted  by  HQ  USAF  to  the  Secretary  of 
Defense,  introduces  a  new  program  to  the  F&FP  or  changes  an  approved 
program  element  in  excess  of  established  thresholds.  A  "proposed  PCP" 
is  submitted  by  AFSC  to  request  an  appropriate  change  to  the  program. 
The  implementing  command  initially  submits  the  PCP  to  the  appropriate 
HQ  USAF  OPR  along  with  a  PTDP,  PSPP,  or  other  technical  backup  data 
attached. 


3.  Preliminary  Technical  Development  Plan  (PTDP) 

This  document  is  submitted  by  AFSC  as  the  initial  response 
to  the  RAD  indicating  approval  of  the  ROC.  The  PTDP  is  used  by  HQ 
USAF  to  support  the  PCP  submitted  to  OSD  for  approval  of  the  definition 
phase. 


4.  Proposed  System  Package  Plan  (PSPP) 

This  document,  normally  prepared  by  AFSC,  is  submitted 
as  a  product  of  the  definition  phase  or  on  direction  of  HQ  USAF.  It  in¬ 
cludes  a  system  description,  cost  estimates,  resource  requirements, 
performance  specifications,  schedules,  and  related  information  for  each 
alternative  proposed.  It  should  be  definitive  enough  to  allow  incentive 
and/or  fixed-price  contracts  to  be  negotiated  in  the  acquisition  phase. 

5.  System  Program  Directive  (SP  Directive) 

This  formal  document,  issued  by  HQ  USAF,  approves  a  sys¬ 
tem  program  defined  in  the  PSPP  and  authorizes  the  publication  of  the 
SPP.  The  SP  Directive  identifies  the  availability  of  financial  and  other 
resources,  the  importance  category,  the  impact  on  other  Air  Force  pro¬ 
grams,  and  other  program  direction.  Subsequent  program  changes  are 
made  as  amendments  to  the  SP  Directive. 
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6. 


System  Definition  Directive  (SDD) 


This  is  the  formal  document  issued  by  HQ  USAF  approving 
the  PTDP.  The  SDD  identifies  the  availability  of  financial  and  other 
resources  as  applicable,  provides  authority  to  AFSC  to  establish  a  for¬ 
mal  SPO,  sets  the  parameters  for  the  System  Program  Director  (SPD), 
and  establishes  the  roles  of  the  participating  organizations.  The  SDD 
also  constitutes  authority  for  solicitation  of  industry  sources  with  the 
intent  to  commit  the  Government  within  approved  fund  authorizations. 

7 .  System  Package  Program  (SPP) 

The  SP  Directive  requires  the  System  Program  Director 
(SPD),  who  is  head  of  the  SPO  and  manager  of  the  approved  system  pro¬ 
gram  during  the  definition  and  acquisition  phases,  to  convert  the  approved 
portions  of  the  PSPP  into  the  SPP.  The  SPP  specifies  the  integrated  and 
time-phased  tasks  and  resources  required  of  and  by  all  participating  or¬ 
ganizations  in  acquiring  and  supporting  the  system. 

A  complete  SPP  consists  of  the  following  sections: 


o 

Section  1 : 

Program  Summary 

o 

Section  2: 

Schedules 

o 

Section  3: 

Program  Management 

o 

Section  4: 

Intelligence  Estimate 

o 

Section  5: 

Operations 

o 

Section  6: 

Acquisition 

o 

Section  7 : 

Civil  Engineering 

o 

Section  8: 

Logistics 

o 

Section  9: 

Manpower  and  Organization 

o 

Section  1  0: 

Personnel  Training 

o 

Section  1  1 : 

Financial 

o 

Section  12: 

Requi  rements 

o 

Section  13: 

Authorizations 

o 

Section  14: 

General  Information 

o 

Section  15: 

Security 

o 

Section  1  6: 

Biomedical 
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In  general,  the  Preliminary  Technical  Development  Plan  (PTDP)  and  the 
Proposed  System  Package  Plan  (PSPP)  contain  the  same  type  of  informa¬ 
tion  and  follow  the  same  order.  Section  14,  General  Information,  must 
include  (AFR  375-4)  a  description  of  all  EDP  systems  used  in  support  of 
the  proposed  system  (but  not  an  integral  part  of  the  system). 

D.  AFR  100  Series  Regulations 

The  100  series  regulations  deal,  in  general,  with  communications - 
electronics  activities  within  the  Air  Force.  In  many  instances,  comput¬ 
ers  are  involved  in  such  systems;  hence  AFADA  becomes  involved  in 
the  approval  cycle  (AFR  300-2A). 

AFR  100-2  defines  a  ground  communications  electronics  meteoro¬ 
logical  (CEM)  system  as  two  or  more  physically  separated  but  interde¬ 
pendent  and  interrelated  equipment  or  facilities,  complete  with  support¬ 
ing  structures  and  services.  Ground  CEM  requirements  can  be  of  two 
types:  quantitative  and  qualitative.  A  quantitative  requirement  is  de¬ 
fined  as  a  need  for  specific  equipment  or  capability  to  accomplish  a  mis¬ 
sion  wherein  the  equipment  or  capability  is  available  without  further  re¬ 
search  and  development  effort.  A  qualitative  requirement  is  defined  as 
a  need  for  a  particular  capability  to  accomplish  a  mission  wherein  the 
equipment  or  techniques  must  be  researched  or  developed. 

A  qualitative  ground  CEM  requirement  is  prepared  and  submitted 
to  HQ  USAF  (AFORQ)  as  an  ROC  (Required  Operational  Capability).  (AFR 
57-3  previously  required  a  QOR,  but  this  regulation  has  been  superseded 
by  AFR  57-1,  17  June  1966.)  After  HQ  USAF  recognizes  and  validates  a 
requirement,  including  OSD  approval,  presumably  an  RAD  is  issued. 

This  document  should  describe  the  characteristics  of  the  required  CEM 
equipment  and  levy  the  requirement  on  AFSC  to  develop  a  new  item  of 
equipment  or  determine  other  means  of  satisfying  the  requirement.  Im¬ 
plementation  will  be  under  AFM  100-18  or  37  5  series  as  directed  by 
HQ  USAF. 

Quantitative  ground  CEM  requirements  are  submitted  to  HQ  USAF 
(AFSME)  for  validation  as  an  Advance  Communications -Electronic  Re¬ 
quirements  Plan  (ACERP)  or  a  Communications -Electronic s  Implemen¬ 
tation  Plan  (CEIP).  If  data  processing  is  involved,  ACERP1  s  and  CEIP's 
are  also  submitted  to  AFADA  and  are  accepted  by  this  organization  in  lieu 
of  DAP's. 

The  ACERP  is  a  statement  of  a  current  or  future  need  for  ground 
CEM  equipment  or  facilities  that  are  available  without  further  develop¬ 
ment  or  research.  Approval  of  an  ACERP  by  HQ  USAF  constitutes  ac¬ 
knowledgement  and  recognition  of  the  stated  operational  requirement 
(approval  in  principle)  and  authorizes  preparing  and  processing  a  CEIP. 

In  certain  instances,  the  ACERP  is  accepted,  CEIP  requirements  are 
waived,  and  AFEC  is  directed  to  implement  the  approved  ACERP. 
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The  CEIP  is  a  detailed  plan  that  provides  information  essential 
for  final  operational  evaluation  and  programming  actions. 

E.  AFR  80-2,  Documents  Used  in  the  Management  of  Air  Force 

Research  and  Development 

AFR  300-7,  Data  Automation,  R&D  Support,  specifically  excludes 
ADP  equipment  developed  for  a  particular  use  through  expenditure  of 
RDT&E  funds.  It  is  therefore  possible  for  computing  equipment  to  be 
acquired  through  submission  of  a  development  plan,  as  described  in 
AFR  80-2,  Attachment  2.  Section  9c  of  these  instructions  requires 
only  a  minimum  of  data  regarding  EDP  equipment. 
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APPENDIX  G 


SUMMARY  OF  CURRENT  REPORTS 
COVERING  AIR  FORCE  ADP  EXPERIENCE 
AND  ASSETS 
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